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VALIDATION  OF  THE  INFORMATION/COMMUNICATIONS  TECHNOLOGY 
LITERACY  TEST 

EXECUTIVE  SUMMARY 


The  United  States  Army  Cyber  Center  of  Excellence  (Cyber  CoE)1  asked  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  assist  in  the  development  of  a 
methodology  to  improve  the  trainee  selection  process.  Specifically,  Cyber  CoE  requested 
information  about  adding  a  cyber-related  aptitude  test  to  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB).  The  joint  service  Information/Communication  Technology  Literacy 
(ICTL)  test  is  a  cognitive  measure  designed  in  the  mold  of  an  ASVAB  technical  subtest  (i.e., 
Automotive  and  Shop  Information,  Electronics  Information,  General  Science,  Mechanical 
Comprehension).  The  ICTL  test  was  designed  to  predict  training  performance  in  cyber-related 
occupations.  Many  Army  Cyber  MOS  have  comparable  duties  to  Air  Force  cyber  occupations 
and  the  Navy’s  Cryptologic  Technician  -  Networks  and  Information  Technologies  occupations, 
for  which  the  ICTL  test  has  shown  evidence  of  validity  in  predicting  cyber- specific  task  or 
knowledge  based  performance  outcomes  such  as  course  grades  and  academic  training  attrition 
(Russell  &  Sellman,  2010;  Trippe  &  Russell,  2011). 

The  purpose  of  the  research  effort  was  to  longitudinally  validate  a  measure  of  cyber  aptitude 
in  predicting  trainee  performance  in  Information  Systems  Operator-Analyst  (25B)  and  Nodal 
Network  Systems  Operator-Maintainer  (25N)  MOS.  This  report  documents  technical  procedures 
and  results  of  the  research  effort. 

The  ICTL  test  was  administered  during  the  first  week  of  training  in  the  Information  Systems 
Operator-Analyst  (25B)  ( n  =1,805)  and  Nodal  Network  Systems  Operator-Maintainer  (25N)  (n  = 
314)  MOS  as  part  of  this  research  effort.  As  Soldiers  neared  the  end  of  training  in  the  focal 
MOS,  they  were  administered  a  battery  of  criterion  assessments  comprising  a  general  job 
knowledge  test,  a  survey  of  attitudes  and  experiences,  peer  ratings  of  MOS -specific  performance 
dimensions,  Warrior  Tasks  and  Battle  Drills  job  knowledge  test  (WTBD  JKT),  and  the  Army 
Life  Questionnaire  (ALQ). 

A  number  of  statistically  significant  relationships  were  observed  between  the  ICTL  and 
important  outcomes  metrics  for  the  25B  MOS.  ICTL  scores  were  significantly  related  to  peer 
ratings  of  MOS  specific  job  performance;  those  with  higher  ICTL  scores  have  higher  peer-rated 
MOS-specific  job  performance  ratings.  The  indication  is  that  the  ICTL  test  is  effective  in 
discriminating  between  low  and  high  performers  in  Advanced  Individual  Training  (AIT).  ICTL 
scores  were  also  significantly  related  to  final  AIT  course  grades,  which  corroborates  the  former 
finding.  ICTL  scores  were  significantly  related  to  a  Soldier’s  likelihood  of  graduating  AIT 
without  an  academic  failure.  That  is,  those  with  higher  ICTL  scores  are  more  likely  to  graduate 
AIT  without  an  academic  failure  than  those  with  lower  ICTL  scores.  ICTL  scores  were 
positively  related  to  perceptions  of  MOS  fit,  indicating  that  the  ICTL  might  function  as  an 


1  At  the  time  of  this  work  was  the  Signal  Center  of  Excellence.  The  Signal  Center  of  Excellence  transitioned  to  the 
Cyber  Center  of  Excellence  in  March  2014. 


indicator  of  interest  and  motivation.  ICTL  scores  were  significantly  related  to  final  AIT  course 
grades  and  perceptions  of  MOS  fit  in  the  25N  MOS. 

The  ICTL  test  provides  appreciable  incremental  validity  beyond  the  AFQT  when  predicting 
the  two  most  job  specific  criteria  (i.e.,  Army  Initial  Training  [AIT]  grades  and  peer  performance 
ratings  scale  [PRS])  in  both  MOS.  ICTL  scores  also  provide  appreciable  incremental  validity 
beyond  aptitude  area  composites  when  predicting  AIT  grades  and  PRS  criteria.  ICTL  scores 
provide  substantial  incremental  validity  beyond  the  Electronics  Information  (El)  test  in 
predicting  all  criteria  (Warrior  Tasks  and  Battle  Drills  job  knowledge  test,  Final  AIT  course 
Grade,  and  Graduate  AIT  without  Failure)  except  the  Performance  Rating  Scales  means  in  25N. 
ICTL  scores  provide  appreciable  incremental  validity  in  predicting  perceptions  of  MOS  fit  in  the 
25B  MOS  as  well.  Indices  of  fairness  (e.g.,  sub-group  differences  and  differential  prediction) 
suggest  than  the  ICTL  test  generally  demonstrates  evidence  of  smaller  disparities  that  those 
observed  in  ASVAB-based  predictors. 

Results  suggest  that  the  ICTL  test  has  potential  as  a  valid  and  highly  efficient  predictor  of 
valued  outcomes  in  Cyber  MOS.  Not  only  is  the  ICTL  test  a  valid  predictor  of  job  knowledge 
and  performance  related  criteria  such  as  course  grades,  but  is  also  a  valid  predictor  of  perceived 
MOS  fit.  This  finding  lends  support  to  the  notion  of  the  ICTL  test  functioning  as  an  indirect 
measure  of  interest,  intrinsic  motivation,  and  skill  in  a  particular  area.  Just  as  the  Automotive  and 
Shop  (AS)  test  can  be  thought  of  as  a  way  to  identify  hobbyists  who  like  to  work  on  cars  or 
motorcycles  and  are  therefore  more  likely  to  perceive  better  fit  in  automotive  related  MOS,  the 
ICTL  is  likely  operating  at  some  level  to  capture  variance  related  to  applicants  in  the  information 
technology  (IT)  domain  who  like  to  do  things  like  build  computers  and  configure  elaborate  home 
networks. 

What  is  perhaps  most  notable  about  the  pattern  of  validity  and  incremental  validity  results  is 
the  ICTL  test’s  efficiency  of  prediction  in  these  Signal  MOS.  In  general,  the  ICTL  test  predicts 
performance  just  as  well  as  composites  derived  from  multiple  ASVAB  tests.  Moreover,  the  ICTL 
test  explains  additional  variance  beyond  these  composites  in  almost  every  criterion  measure. 
Validity  of  the  ICTL  test  is  substantially  greater  than  its  closest  counterpart  in  the  ASVAB,  the 
El  test,  in  predicting  performance  in  these  particular  MOS.  Thus  it  represents  a  useful 
supplement  to  ASVAB  for  cyber  occupations. 


IV 


VALIDATION  OF  THE  INFORMATION/COMMUNICATIONS  TECHNOLOGY 
LITERACY  TEST 

CONTENTS 


Page 


INTRODUCTION . 1 

Background  on  Development  of  the  ICTL  Test  2 

METHOD . 3 

ICTL  Administration  at  Signal  School  3 

Criterion  Measure  Adaptation  &  Development  4 

Job  Knowledge  Test  (JKT) . 4 

Army  Life  Questionnaire  (ALQ) 4 

Performance  Rating  Scales  (PRS) . 7 

Administrative  Data  8 

RESULTS . 9 

ICTL  Score  Relationships  with  Criterion  Measures  9 

Incremental  V alidity  18 

Lairness  Analyses  24 

DISCUSSION . 28 

SUMMARY  AND  CONCLUSION . 28 

RELERENCES . 29 

APPENDIX  A-l 


LIST  OF  TABLES 


Table  1.  Summary  of  ICTL  Scores  by  MOS  3 

Table  2.  Summary  of  JKT  Scores  by  MOS  4 

Table  3.  ALQ  Likert-Type  Scales . 5 

Table  4.  Summary  of  Relevant  ALQ  Scale  Scores  by  MOS  6 

Table  5.  CLA  Model  Lit  Indices  for  the  ALQ  7 

Table  6.  Summary  of  Peer  PRS  by  MOS  8 

Table  7.  CPA  Model  Pit  Indices  for  a  General  Pactor  in  the  PRS  8 


v 


CONTENTS  (continued) 


Page 

Table  8.  Summary  of  Relevant  ASVAB  Scores  by  MOS  9 

Table  9.  Summary  of  Administrative  Criteria  9 

Table  10.  Bivariate  Correlations  between  ICTL  and  Relevant  Criteria  10 

Table  11.  Incremental  Validity  of  ICTL  in  Predicting  Job  Knowledge/Performance 

Criteria  20 

Table  12.  Incremental  Validity  of  ICTL  in  Predicting  Fit  and  Retention  Criteria  20 

Table  13.  Incremental  Validity  of  ICTL  in  Predicting  Job  Knowledge/Performance 

Criteria  Corrected  for  Multivariate  Range  Restriction  on  the  ASVAB  21 

Table  14.  Incremental  Validity  of  ICTL  in  Predicting  Fit  and  Retention  Criteria  Corrected 

for  Multivariate  Range  Restriction  on  the  ASVAB  21 

Table  15.  Gender  and  Racial/Ethnic  Subgroup  Means,  Standard  Deviations,  and 

Standardized  Group  Mean  Differences  (Cohen’s  d)  among  Predictor  and 
Criterion  Variables  in  25B  and  25N  MOS  26 

Table  16.  Gender  and  Racial/Ethnic  Subgroup  Criterion-related  Validity  Estimates  and 

Moderated  Multiple  Regression  Results  for  Differential  Prediction  Analyses  27 


LIST  OF  FIGURES 

Page 

Figure  1.  Expectancy  charts  for  fit  and  retention  outcomes  in  the  25B  MOS.  12 

Figure  2.  Expectancy  charts  for  perception  of  MOS  fit  in  the  25B  MOS.  13 

Figure  3.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance 

outcomes  in  the  25B  MOS.  14 

Figure  4.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance 

outcomes  in  the  25B  MOS.  15 

Figure  5.  Expectancy  charts  for  fit  and  retention  outcomes  of  MOS  fit  in  the  25N  MOS.  16 

Figure  6.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance 

outcomes  in  the  25N  MOS.  17 

Figure  7.  Incremental  validity  of  ICTL  in  predicting  job  knowledge/performance  criteria.  22 

Figure  8.  Incremental  validity  of  ICTL  in  predicting  job  knowledge/performance  criteria.  23 


vi 


VALIDATION  OF  THE  INFORMATION/COMMUNICATIONS 
TECHNOLOGY  LITERACY  TEST 


Introduction 

The  Unites  States  Army  Cyber  Center  of  Excellence  (Cyber  CoE)  asked  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  assist  in  the  development  of  a 
selection  tool  to  improve  the  trainee  selection  process.  Specifically,  Cyber  CoE  requested 
information  about  adding  a  cyber  specific  aptitude  test  to  compliment  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB).2 

The  Information/Communication  Technology  Literacy  (ICTL)  test  is  a  cognitive  measure 
designed  in  the  mold  of  an  ASVAB  technical  subtest.  The  ICTL  test  was  developed  and 
validated  by  the  Air  Force,  with  all  the  Services  contributing,  to  predict  training  performance  in 
cyber-related  occupations.  Many  Army  Signal  MOS  have  comparable  duties  to  Air  Force  cyber 
occupations.  For  the  Navy’s  Cryptologic  Technician  -  Networks  (CTN)  and  Information 
Technologies  (IT)  occupations,  the  ICTL  test  has  shown  evidence  of  validity  in  predicting  cyber¬ 
specific  task  or  knowledge  based  performance  outcomes  such  as  course  grades  and  academic 
training  attrition  (Russell  &  Sellman,  2010;  Trippe  &  Russell,  2011). 

The  ICTL  test  may  also  function  well  as  an  indirect  indicator  of  MOS  fit  or  motivation-based 
performance  outcomes.  Similar  to  ASVAB  technical  subtests,  the  ICTL  measure  is  an 
information  test.  Information  tests  were  among  the  most  successful  and  most  highly  valid 
classification  tests  created  by  the  Army  Air  Force's  (AAF)  Aviation  Psychology  Program  during 
World  War  II.  Guilford  and  Lacey  (1947)  described  the  logic  of  information  tests  as  follows: 

It  is  becoming  recognized  more  and  more  that  what  a  person  knows  or  does  not 
know  can  be  used  to  reveal  a  number  of  things  concerning  his  personal 
background.  Since  he  is  to  a  large  extent  a  product  of  his  personal  experience,  and 
since  what  he  is  bodes  good  or  ill  concerning  his  future  status  in  one  respect  or 
another,  knowledge  scores  promise  to  have  predictive  value  (p.  341). 

The  key  notion  is  that  information  tests  are  thought  to  be  indirect  measures  of  interest, 
intrinsic  motivation,  and  skill  in  a  particular  area.  Although  the  ICTL  test  is  a  cognitive  measure, 
it  is  likely  to  have  the  strongest  relationship  with  cyber- specific  tasks  or  knowledge-based 
performance  outcomes  such  as  course  grades.  As  such,  it  is  reasonable  to  hypothesize  that  the 
ICTL  will  correlate  with  attitudes  related  to  occupational  fit. 

The  purpose  of  the  research  effort  was  to  longitudinally  validate  a  measure  of  cyber  aptitude 
in  predicting  trainee  performance  in  Information  Systems  Operator- Analyst  (25B)  and  Nodal 
Network  Systems  Operator-Maintainer  (25N)  MOS.  We  summarize  the  adaptation  and 
development  of  criterion  measures  and  present  results  of  psychometric  and  predictive  validity 
analyses. 


2  ASVAB  tests/composites  include:  Arithmetic  Reasoning  (AR),  Assembling  Objects  (AO),  Auto  &  Shop 
Information  (AS),  Electronics  Information  (El),  General  Science  (GS),  Math  Knowledge  (MK),  Mechanical 
Comprehension  (MC),  Paragraph  Comprehension  (PC),  and  Word  Knowledge  (WK). 
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Background  on  Development  of  the  ICTL  Test 

In  2005-2006,  the  Department  of  Defense  (DoD)  convened  a  panel  of  experts  in  the  areas  of 
personnel  selection,  job  classification,  psychometrics,  and  cognitive  psychology  to  provide 
recommendations  for  improving  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB). 

The  panel  made  22  recommendations  regarding  test  content  specifications,  administration, 
validation  procedures,  and  new  test  content  areas.  One  of  the  review  panel’s  recommendations 
stated  that  “research  should  be  conducted  to  develop  and  evaluate  a  test  of  information  and 
communications  technology  literacy.  The  efficacy  of  coaching  and  item  familiarity,  as  well  as 
the  feasibility  of  creating  multiple  forms,  should  be  examined  in  conjunction  with  test 
development”  (Drasgow,  Embretson,  Kyllonen,  &  Schmidt,  2006,  p.  26). 

Toward  that  end,  the  U.S.  Air  Force  assumed  responsibility  as  the  lead  organization  in 
development  of  an  ICTL  test  which  could  potentially  be  added  to  the  ASVAB.  The  first  phase  of 
the  research,  to  develop  and  pilot  test  an  ICTL  measure,  was  conducted  in  FY  2008.  The  specific 
objectives  were  to  (a)  prepare  a  content  blueprint  indicating  what  the  test  should  measure,  (b) 
develop  and  pilot  a  draft  version  of  the  test,  (c)  assemble  new  test  forms,  and  (d)  plan  validation 
research.  The  test  had  three  components:  (a)  background  information,  or  biodata,  (b) 
information-communications  technology  knowledge,  and  (c)  logic.  Based  on  the  results  of  the 
pilot  test,  pre-equated  alternate  forms  of  the  ICTL  were  developed. 

The  purpose  of  Phase  II  was  to  assess  the  validity  of  the  ICTL  measure  for  predicting  success 
in  technical  training.  Seven  Air  Force  technical  training  schools  (e.g.,  Communications, 

Network,  Switch  &  Crypto  Systems)  and  two  Navy  “A”  schools3  (i.e.,  Information  Systems 
Technician  and  Cryptologic  Technician  [Networks])  participated  in  the  project.  All  but  two  of 
the  occupations  included  were  cyber  occupations.  Non-cyber  occupations  provided  an 
opportunity  to  evaluate  discriminant  validity.  A  predictor  battery  including  the  ICTL  test,  a 
biodata  measure,  and  a  figural  reasoning  test  (a  measure  of  nonverbal  reasoning)  was 
administered  to  students  at  the  beginning  of  class.  Final  school  grades  (FSGs)  were  collected  to 
serve  as  criteria  for  validating  the  measures.  In  total,  1,396  students  had  complete  predictor  data 
and  FSGs.  The  ICTL  measure  predicted  FSGs  significantly  for  all  but  one  of  the  cyber 
occupations.  It  was  a  significant  predictor  in  one  of  the  non-cyber  occupations  (Security  Forces) 
as  well.  Analyses  also  suggested  that  the  ICTL  was  a  better  predictor  than  Electronics 
Information  (El),  one  of  the  ASVAB  subtests  currently  included  in  composites  used  to  select 
military  applicants  for  many  of  the  cyber  occupations  (Russell  &  Sellman,  2010).  Phase  II 
indicated  that  additional  further  research  was  warranted. 

The  primary  objectives  of  Phase  III  were  to  assess  functioning  of  the  test  in  an  applicant 
population  and  to  develop  operational  test  forms.  One  of  four  40-item  experimental  forms  was 
administered  to  52,708  military  service  applicants  at  Military  Entrance  Processing  Station 
(MEPS)  in  a  randomly  equivalent  groups  common  item  design.  Once  analyses  of  the  MEPS 
forms  were  complete,  two  29-item  operational  forms  were  created.  The  forms  are  equivalent 
with  respect  to  content  balance,  difficulty,  discrimination  and  reliability  (Trippe  &  Russell, 


3  The  Navy  calls  its  job  training  “A”  school.  All  Navy  enlisted  ratings  (jobs)  have  an  A  school,  which  teaches  the 
fundamentals  of  the  specific  Navy  job. 
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2011).  Analyses  of  the  operational  test  form  scores  showed  that  the  ICTL  test  exhibits  smaller 
subgroup4  differences  than  the  ASVAB  technical  knowledge  tests. 

A  second  objective  of  Phase  III  was  to  determine  how  well  the  ICTL  test  predicts  success  in 
technical  training  for  the  Navy’s  Cryptologic  Technician  (Networks),  or  CTN,  school.  Phase  II 
data  showed  that  the  ICTL  test  scores  significantly  predicted  performance  in  CTN  school; 
however,  near  the  conclusion  of  that  study,  the  Navy  altered  the  CTN  course  format.  Phase  III 
data  collected  at  the  CTN  school  (n  =  118)  for  about  a  year  showed  that  the  ICTL  test  was  a 
significant  predictor  of  both  grade  point  average  and  graduation  status  (i.e.,  graduated  vs.  did  not 
graduate)  in  the  new  course  format.  The  ICTL  test  also  provided  significant  incremental  validity 
over  CTN  school  selection  composites  (Trippe  &  Russell,  2011). 

Method 


ICTL  Administration  at  Cyber  CoE  School 

The  current  operational  forms  of  the  ICTL  test  were  administered  via  computer  during  the 
first  week  of  advanced  individual  training  (AIT)  in  the  Information  Systems  Operator- Analyst 
(25B)  and  Nodal  Network  Systems  Operator-Maintainer  (25N)  MOS  as  part  of  this  research 
effort  (25B  n  =1,805;  25N  n  =  314).  One  of  two  parallel  29-item  forms  was  randomly  assigned  to 
each  Soldier.  Five  groups  were  examined:  males  (25B  n  =  1,371;  25N  n  =  254),  females  (25B  n 
=  359;  25N  n  =  39),  non-Hispanic  Blacks  (25B  n  =  522;  25N  n  =  62),  non-Hispanic  Whites  (25B 
n  =  803;  25N  n  =  164),  and  Hispanic  Whites  (25B  n  =  247;  25N  n  =  37).  Table  1  presents  ICTL 
test  scores  by  MOS  in  both  the  scaled  reporting  and  percent  correct  metric.  The  scaled  scores  are 
an  Item  Response  Theory-based  maximum  a  posteriori  (MAP)  ability  estimate  that  has  been 
placed  on  an  adjusted  t-score  scale.  MAP  estimation,  Bayes  modal  estimation,  considers  the 
examinee’s  pattern  of  item  responses  in  relation  to  a  set  of  item  parameters  that  characterize  the 
difficulty,  discrimination  and  guessing  potential  of  each  item  as  well  as  an  assumed  distribution 
of  ability  (Embretson  &  Reise,  2000).  MAP  ability  estimates  were  computed  using  the 
commercial  software  MULTILOG  (Thissen,  2003).  A  standard  t-score  distribution  has  a  mean  of 
50  and  standard  deviation  of  10.  The  ICTL  reporting  metric  has  been  adjusted  such  that  the 
standard  distribution  would  be  expected  in  the  youth  population  (Profile  of  American  Youth 
[PAY97]  sample;  DMDC,  2003).  Scaled  ICTL  scores  are  used  in  the  validation  analyses 
reported  below  to  reflect  the  use  of  scaled  scores  in  operational  decision  making.  Scores  were  not 
assigned  to  Soldiers  who  omitted  more  than  five  test  items  or  completed  the  assessment  in  less 
than  three  minutes. 


4Subgroup  comparisons  were  male  vs.  female,  non-Hispanic  White  vs.  non-Hispanic  Black  and  non-Hispanic  White 
vs.  Hispanic  White.  These  groups  were  chosen  to  be  consistent  with  designations  used  by  the  ASVAB  testing 
program  (Defense  Manpower  Data  Center,  2011) 
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Table  1 

Summary  ofICTL  Scores  by  MOS 


Score 

25B  (n 

=1.805) 

25  N  (n 

=  314) 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

ICTL  Scaled  Score 

55.3 

8.1 

27 

79 

59.9 

7.7 

35.0 

79.0 

ICTL  %  Correct 

58.6 

14.9 

17.2 

100 

66.7 

14.0 

31.0 

100 

Criterion  Measure  Adaptation  and  Development 

As  Soldiers  neared  the  end  of  training  in  the  focal  MOS,  they  were  administered  via 
computer  a  battery  of  criterion  assessments  comprising  a  general  job  knowledge  test,  a  survey  of 
attitudes  and  experiences,  and  peer  ratings  of  MOS-specific  performance  dimensions.  Each 
criterion  measure  is  described  in  more  detail  below. 

Job  Knowledge  Test  (JKT). 

The  Warrior  Tasks  and  Battle  Drills  job  knowledge  test  (WTBD  JKT)  was  administered  to 
all  Soldiers  participating  in  this  research  effort.  The  WTBD  JKT  measures  knowledge  that  is 
general  to  all  enlisted  Soldiers  and  includes  a  mix  of  item  formats  (e.g.,  multiple-choice  and 
multiple-response).  The  items  use  visual  images  to  make  them  more  realistic  and  reduce  reading 
requirements  for  the  test.  The  WTBD  JKT  was  developed  as  part  of  a  separate  research  project 
(Knapp  &  Heffner,  2010).  Prior  to  finalizing  the  items,  in  the  summer  of  201 1,  for  use  in  that 
project,  the  items  were  reviewed  by  project  staff  and  Army  subject  matter  experts  (SMEs)  to 
ensure  they  were  of  high  quality.  Poorly  performing  or  outdated  items  were  replaced,  and 
additional  items  were  included  to  ensure  adequate  coverage  of  content  areas  identified  in  the  test 
blueprints  that  had  been  established  for  the  test. 

JKT  scores  were  flagged  as  invalid  if  the  Soldier  (a)  omitted  more  than  10%  of  the 
assessment  items,  (b)  took  fewer  than  5  minutes  to  complete  the  entire  assessment,  or  (c) 
selected  an  implausible  response  to  one  of  the  embedded  careless  responding  items.  Table  2 
contains  a  summary  of  the  valid  WTBD  JKT  scores  by  MOS.  Cronbach’s  coefficient  alpha, 
which  is  an  internal  consistency  index  of  reliability,  is  .72  in  the  combined  MOS  sample. 


Table  2 

Summary  of  JKT  Scores  by  MOS 


Score 

25  B  (n  = 

=959) 

25  N  (n  = 

=  146) 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

WTBD  %  Correct 

61.3 

11.8 

13.5 

94.6 

66.0 

10.3 

29.7 

86.5 

Army  Life  Questionnaire  (ALQ). 

The  ALQ  was  designed  to  measure  Soldiers’  self-reported  attitudes  and  experiences  in  the  Army. 
The  ALQ  includes  scales  that  cover  (a)  Soldiers’  commitment  and  retention-related  attitudes  and  (b) 
Soldiers’  performance  and  adjustment.  Each  ALQ  scale  is  scored  differently  depending  on  the  nature 
of  the  attribute  being  measured.  The  Army  Physical  Fitness  Test  (APFT)  score  is  a  write-in  item. 
Training  Achievements,  Training  Failures  and  Disciplinary  Incidents  are  simply  a  sum  of  the  “yes” 
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responses.  The  remaining  scales  (see  Table  3)  are  scored  with  Likert-type  scales  by  computing  a 
mean  of  the  constituent  item  scores  after  accounting  for  reverse  coded  items. 


Table  3 

ALQ  Likert-Type  Scales 


Scale  Name 

Description 

Number 
of  Items 

Example  Item 

Likert  Scale  Anchors 

Affective 

Commitment 

Measures  Soldiers’ 
emotional  attachment  to 
the  Army. 

7 

I  feel  like  I  am  part  of  the 
Army  ‘family.’ 

1  (strongly  disagree)  to  5 
(strongly  agree) 

Normative 

Commitment 

Measures  Soldiers’  feelings 
of  obligation  toward 
staying  in  the  Army  until 
the  end  of  their  current 
term  of  service. 

5 

I  would  feel  guilty  if  I  left  the 
Army  before  the  end  of  my 
current  term  of  service. 

1  (strongly  disagree)  to  5 
(strongly  agree) 

Career 

Intentions 

Measures  Soldiers’ 
intentions  to  reenlist  and  to 
make  the  Army  a  career. 

3 

How  likely  is  it  that  you  will 
make  the  Army  a  career? 

Varies  by  item:  1  (strongly 
disagree)  to  5  (strongly 
agree);  1  (not  at  all 
confident)  to  5  (extremely 
confident);  1  (extremely 
unlikely  to  5  (extremely 
likely) 

Reenlistment 

Intentions 

Measures  Soldiers’ 
intention  to  reenlist  in  the 
Army. 

4 

I  intend  to  leave  the  Army 
after  completing  my  current 
term  of  service. 

1  (strongly  disagree)  to  5 
(strongly  agree) 

Attrition 

Cognitions 

Measures  the  degree  to 
which  Soldiers  think  about 
attriting  before  the  end  of 
their  first  term. 

4 

I  am  confident  that  I  will 
complete  my  current  term  of 
service. 

Varies  by  item:  1  (strongly 
disagree)  to  5  (strongly 
agree);  1  (never)  to  5  (very 
often) 

Army  Life 
Adjustment 

Measures  Soldiers’ 
transition  from  civilian  to 
Army  life. 

9 

Looking  back,  I  was  not 
prepared  for  the  challenges  of 
training  in  the  Army. 

1  (strongly  disagree)  to  5 
(strongly  agree) 

Army  Civilian 
Comparison 

Measures  Soldiers’ 
impressions  of  how  Army 
life  compares  to  civilian 
life. 

6 

Indicate  how  you  believe 
conditions  in  the  Army 
compare  to  conditions  in  a 
civilian  job  with  regards  to 
pay  and  other  factors  (e.g., 
advancement  opportunities, 
job  security). 

1  (much  better  in  the  Army) 
to  5  (much  better  in  civilian 
life) 

MOS  Fit 

Measures  Soldiers’ 
perceived  fit  with  their 
MOS. 

9 

My  MOS  provides  the  right 
amount  of  challenge  for  me. 

1  (strongly  disagree)  to  5 
(strongly  agree) 

Army  Fit 

Measures  Soldiers’ 
perceived  fit  with  the 

Army. 

8 

The  Army  is  a  good  match 
for  me. 

1  (strongly  disagree)  to  5 
(strongly  agree) 

As  with  the  JKT,  ALQ  data  were  flagged  as  unusable  if  the  Soldier  (a)  omitted  more  than 
10%  of  the  assessment  items,  (b)  took  fewer  than  5  minutes  to  complete  the  entire  assessment,  or 
(c)  chose  an  implausible  response  to  the  embedded  careless  responding  item.  Table  4  contains  a 
summary  of  potentially  relevant  ALQ  scales  scores  (i.e.,  those  scales  the  ICTL  test  might 
reasonably  be  hypothesized  to  predict)  by  MOS.  A  summary  of  the  distributions  of  all  ALQ  scale 
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scores  as  well  as  their  relationship  with  predictor  variables  can  be  found  in  Table  A3  of  the 
Appendix. 


Table  4 

Summary  of  Relevant  ALQ  Scale  Scores  by  MOS 


ALQ  Scale 

25  B  (n 

=  1012) 

25N  (n 

=  153) 

a 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

Army  Fit 

.86 

4.0 

0.6 

1.0 

5.0 

3.9 

0.6 

2.1 

5.0 

Attrition  Cognitions 

.75 

1.6 

0.6 

1.0 

4.0 

1.7 

0.7 

1.0 

4.8 

Career  Intentions 

.91 

3.1 

1.1 

1.0 

5.0 

2.9 

1.0 

1.0 

5.0 

MOS  Fit 

.93 

3.9 

0.8 

1.1 

5.0 

3.6 

0.8 

1.1 

5.0 

Reenlistment  Intentions 

.81 

3.4 

0.9 

1.0 

5.0 

3.3 

0.9 

1.0 

5.0 

Training  Achievements 

- 

0.3 

0.5 

0.0 

2.0 

0.2 

0.4 

0.0 

1.0 

Training  Failures 

- 

0.5 

0.7 

0.0 

3.0 

0.5 

0.7 

0.0 

3.0 

The  ALQ  conceptual  measurement  model  was  evaluated  in  a  confirmatory  factor  analysis 
(CFA)  framework.  CFA  is  a  component  of  a  larger  paradigm  of  analyses  commonly  known  as 
covariance  structure  analysis  or  structural  equation  modeling  (see  Bollen,  1989).  CFA  allows  the 
user  to  specify  an  a  priori  measurement  model  (by  constraining  parameters  of  the  model),  in 
which  the  relationship  between  observed  (i.e.,  survey  items)  and  latent  (i.e.,  constructs)  variables 
is  hypothesized.  The  covariance  matrix  implied  by  the  hypothesized  model  is  evaluated  against 
the  observed  data  matrix,  thereby  allowing  quantification  of  model  fit. 

Three  measurement  models  were  tested: 

•  A  one-factor  model  in  which  all  ALQ  items  are  explained  by  a  general  factor. 

•  A  four-factor  model  in  which  Army  Fit,  Attrition  Cognitions  and  MOS  fit  items  are  all 
explained  by  their  respective  latent  constructs  and  the  Career  Intentions  and  Reenlistment 
Intentions  items  are  collapsed  and  explained  by  a  fourth  factor. 

•  A  five-factor  model  in  which  Army  Fit,  Attrition  Cognitions,  MOS  fit,  Career  Intentions 
and  Reenlistment  Intentions  items  are  all  explained  by  their  respective  latent  constructs. 

Model  fit  indices  for  the  three  ALQ  models  are  reported  in  Table  5.  The  associated  chi- 
square  values  with  all  models  are  not  statistically  significant;  this  indicates  a  poor  model  fit. 
Nevertheless,  the  chi-square  test  is  not  generally  relied  on  as  an  index  of  overall  model  fit  in 
models  tested  on  samples  larger  than  200  (Kenny,  2009).  Comparative  Fit  Index  (CFI)  and 
Tucker- Lewis  Index  (TLI)  values  above  .95  are  generally  indicative  of  good  model  fit  (higher  is 
better).  Root  Mean  Residual  (RMR)  values  below  .05  and  Root  Mean  Square  Error  of 
Approximation  (RMSEA)  values  below  .08  are  generally  indicative  of  good  model  fit  (lower  is 
better;  Kenny,  2009).  Fit  index  values  in  Table  5  suggest  that  the  single-factor  model  exhibits 
very  poor  fit  and  both  the  four-  and  five-factor  models  fit  the  data  moderately  well.  Because  the 
models  are  nested,  it  is  possible  to  make  direct  statistical  comparisons  using  the  difference 
between  chi-square  values.  The  five-factor  model  fits  the  data  significantly  better  (at  the  .01 
level)  than  the  four-factor  model,  but  the  CFI,  RMSEA,  and  SRMR  values  are  nearly  identical 
between  the  four-  and  five-  factor  models. 
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Table  5 

CFA  Model  Fit  Indices  for  the  ALQ 


Index 

One  Factor 

Four  Factor 

Five  Factor 

Chi-Square 

12329.924 

2555.869 

2533.767 

DF 

350 

344 

340 

CFI 

.459 

.900 

.901 

TLI 

.416 

.890 

.890 

SRMR 

.169 

.057 

.057 

RMSEA 

.166 

.072 

.072 

A  Chi-square 

- 

9774.055 

22.102 

A  DF 

- 

6 

4 

Performance  Rating  Scales  (PRS). 

Peer  performance  rating  scales  (PRS)  were  developed  through  workshops  with  AIT 
instructors  from  each  MOS.  The  first  workshops,  conducted  in  person,  identified  performance 
dimensions  suitable  for  the  training  scales  and  obtained  behavioral  descriptions  of  performance 
within  each  dimension.  Subsequently,  SMEs  also  reviewed  the  products  of  the  workshops.  The 
primary  means  for  this  review  was  a  retranslation  exercise,  which  asked  the  SMEs  to  sort  the 
behavioral  examples  into  the  dimensions.  The  post  exercise  discussion  provided  a  systematic 
way  to  evaluate  the  quality  and  completeness  of  the  behavioral  examples.  Based  on  feedback 
from  the  SMEs,  the  dimensions  and  behavioral  examples  were  further  modified  and  developed 
into  draft  training  PRS  in  preparation  for  the  next  SME  meeting. 

The  next  SME  workshop,  conducted  via  teleconference,  involved  the  SMEs  thinking  of  two 
Soldiers  they  had  in  training  recently  and  rating  these  Soldiers  on  the  draft  PRS.  The  “try-out” 
discussion  led  to  some  minor  wording  changes  and  confirmed  the  instructions  were  clear  and,  for 
the  most  part,  Soldiers  had  ample  opportunity  to  observe  the  behaviors  depicted  in  the  scales. 

The  final  PRS  include  a  “Not  applicable”  or  “Not  observed”  response  option  for  each  scale.  The 
final  peer  PRS  can  be  found  in  Appendix  Tables  A1  and  A2.  The  25B  PRS  comprise  6  scales 
and  the  25N  comprise  8  scales. 

Peer  rating  assignments  were  made  according  to  a  protocol  administered  by  course 
instructors.  According  to  the  protocol,  Soldiers  are  divided  randomly  into  groups  of  a  minimum 
size  of  four  within  each  training  course.  Each  Soldier  has  the  opportunity  to  rate  at  least  three  of 
his  or  her  randomly  assigned  peers  on  each  of  the  MOS-specific  dimensions.  The  PRS 
assessment  also  includes  a  4-point  “familiarity”  rating  in  which  the  rater  indicates  his  or  her 
general  opportunity  to  observe  each  Soldier  being  rated  (i.e.,  “not  enough”  through  “enough  to 
judge  most  aspects  of  performance”).  Based  on  their  familiarity  rating,  each  Soldier  may  then 
rate  all  three  or  none  of  their  peers  assigned  to  them.  Soldiers  in  25B  and  25N  MOS  were  rated 
by  an  average  of  2.7  and  2.9  raters,  respectively.  An  aggregate  peer  rating  is  computed  for  each 
Soldier  as  the  average  of  all  peer  ratings  where  familiarity  was  rated  as  sufficient  to  judge  at  least 
“some  aspects”  of  the  ratee’s  performance.  Table  6  contains  summary  statistics  for  the  peer  PRS 
by  MOS.  Cronbach’s  alpha,  an  index  of  internal  consistency  reliability,  is  .96  across  all  scales 
for  both  25B  and  25N  PRS.  Interrater  reliability  (IRR)  estimates  range  from  .27  to  .60. 
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Table  6 

Summary  of  Relevant  Peer  PRS  by  MOS 


PRS  Scale 

25B  (n  =1,076  ratees) 

25N  (n  = 

169  ratees) 

IRR 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

Implement  Network 

.56 

3.9 

0.9 

1.0 

5.0 

- 

- 

- 

- 

Hardware  Concepts 

.50 

4.0 

0.9 

1.0 

5.0 

- 

- 

- 

- 

Software  Applications 

.43 

4.1 

0.8 

1.0 

5.0 

- 

- 

- 

- 

Network  Security 

.43 

3.9 

0.9 

1.0 

5.0 

- 

- 

- 

- 

Troubleshooting* 

.44 

3.9 

0.9 

1.0 

5.0 

- 

- 

- 

- 

Safety  Procedures* 

.27 

4.1 

0.8 

1.0 

5.0 

- 

- 

- 

- 

Configure  Devices 

.60 

- 

- 

- 

- 

3.9 

0.8 

1.0 

5.0 

Troubleshooting* 

.58 

- 

- 

- 

- 

3.9 

0.7 

1.0 

5.0 

COMSEC 

.41 

- 

- 

- 

- 

4.0 

0.6 

1.3 

5.0 

Network  Architecture 

.57 

— 

— 

— 

- 

4.1 

0.7 

1.0 

5.0 

Device  Access 

.48 

- 

- 

- 

- 

4.1 

0.7 

1.0 

5.0 

Access  Method 

.43 

— 

— 

— 

- 

4.1 

0.6 

1.0 

5.0 

Internet  Security 

.43 

- 

- 

- 

- 

4.0 

0.7 

1.3 

5.0 

Safety  Procedures* 

.35 

- 

- 

- 

- 

4.2 

0.6 

1.7 

5.0 

Peer  Rating  Mean 

.531.59 

4.0 

.78 

1.0 

5.0 

4.0 

.63 

1.2 

5.0 

IRR  =  Interrater  reliability.  Interrater  reliability  was  assessed  using  G(q,k),  a  reliability  metric  designed  specifically  for  studies 
where  the  measurement  design  is  ill-structured  (Putka,  Le.  McCloy,  &  Diaz,  2008). 

*Note  that  Troubleshooting  and  Safety  dimensions  are  defined  differently  according  to  the  demands  of  each  MOS  and  are 
therefore  reported  separately. 

Table  7  contains  CFA  model  fit  indices  for  a  general  (single)  factor  PRS  model  tested  in  each 
MOS.  Model  fit  indices  suggest  that  the  data  fit  a  general  factor  model  very  well  in  the  25B 
MOS  and  fit  is  good  in  the  25N  MOS.  Given  the  high  degree  of  internal  correspondence 
suggested  by  both  the  Cronbach’s  alpha  (i.e.,  >.95)  and  CFA,  only  the  overall  PRS  mean  will  be 
analyzed  in  predictive  analyses  described  later  in  this  report. 

Table  7 

CFA  Model  Fit  Indices  for  a  General  Factor  in  the  PRS 


Index 

25  B 

25  N 

Chi-Square 

17.211* 

67.991** 

DF 

9 

20 

CFI 

.993 

.966 

TLI 

.988 

.952 

SRMR 

.011 

.026 

RMSEA 

.073 

.119 

><.05 

*><.01 


Administrative  Data. 

ASVAB  standard  scores  were  extracted  from  the  Military  Entrance  Processing  Command 
(MEPCOM)  Integrated  Resource  System  (MIRS)  database.  Table  8  contains  summary  statistics  on 
relevant  ASVAB  scores  by  MOS.  The  Armed  Forces  Qualification  Test  (AFQT)  is  included  because 
it  is  a  good  indicator  of  general  mental  aptitude  and  used  for  selection  into  the  Services.  The 


Electronics  Information  (El)  test  is  included  because  it  is  the  closest  counterpart  to  the  ICTL  test  in 
the  extant  ASVAB  battery.  The  Electrical  (EL),  Skilled  Technical  (ST)  and  Surveillance  and 
Communications  (SC)  aptitude  area  composites  are  included  because  they  are  currently  used  for 
Signal  MOS  qualification. 


Table  8 

Summary  of  ASVAB  Scores  by  MOS 


ASVAB 

25B  (n 

=  1,746) 

25N  (n 

=  294) 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

AFQT  Percentile 

63.6 

16.5 

21.0 

99.0 

74.7 

12.9 

40.0 

99.0 

Electronics  Information  (El) 

52.0 

8.5 

23.0 

82.0 

56.6 

7.1 

38.0 

79.0 

Electrical  Comp  (EL)* 

106.8 

11.4 

83.0 

156.0 

115.0 

oo 
o 6 

97.0 

144.0 

Skilled  Tech  Comp  (ST)* 

107.6 

10.8 

85.0 

155.0 

115.5 

8.6 

98.0 

144.0 

Surv.  &  Comm.  Comp  (SC)* 

107.8 

11.0 

85.0 

155.0 

115.8 

8.6 

98.0 

144.0 

*  Aptitude  area  composites  are  weighted  combinations  of  the  following  ASVAB  tests  and  composites:  Arithmetic  Reasoning, 
Auto  &  Shop,  Electronics  Information,  General  Science,  Mechanical  Comprehension,  and  Verbal. 


Data  on  Initial  Military  Training  (IMT)  school  performance  and  completion  were  extracted 
from  (a)  Army  Training  Requirements  and  Resources  System  (ATRRS)  database  produced  by  the 
Training  and  Doctrine  Command  (TRADOC)  and  (b)  Army  Training  Support  Center’s  (ATSC) 
Resident  Individual  Training  Management  System  (RITMS)  data  files.  ATRRS  course  information 
was  used  to  determine  if  Soldiers  graduated  from  AIT  with  or  without  at  least  one  academic 
failure.  Soldiers’  final  AIT  course  grades  were  extracted  from  RITMS.  Table  9  contains  a 
summary  of  these  administrative  criterion  variables.  The  average  final  course  grade  (reported  in 
a  percent  correct  metric)  is  81.8  and  92.4  for  25B  and  25N,  respectively.  Eighty  six  percent  of 
25B  Soldiers  in  the  available  sample  graduated  ATT  without  an  academic  failure  and  90%  of  25N 
Soldiers  in  the  available  sample  graduated  AIT  without  a  failure. 


Table  9 

Summary  of  Administrative  Criteria 


Admin  Criterion 

25B 

25N 

n 

M 

SD 

Min 

Max 

n 

M 

SD 

Min 

Max 

Final  AIT  Course  Grade 

524 

81.8 

9.6 

35.0 

100 

159 

92.4 

5.0 

77.4 

100 

Grad  AIT  w/o  Failure 

1,435 

0.86 

0.35 

0 

1 

228 

0.91 

0.29 

0 

1 

Results 

ICTL  Score  Relationships  with  Criterion  Measures 

Table  10  presents  bivariate  correlations  between  the  ICTL  scaled  scores  and  potentially 
relevant  criterion  measures.  Full  observed  correlation  matrices  can  be  found  in  Tables  A4  and 
A5.  Table  10  also  contains  bivariate  correlations  corrected  for  multivariate  range  restriction  on 
the  ASVAB  subtests  (Lawley,  1943)  using  a  large  sample  ( n  =  483,737)  of  Army  applicants  as 
the  unrestricted  reference  (see  Knapp  &  LaPort,  2014  for  details  of  the  sample).  Statistical 
corrections  for  range  restriction  in  the  predictor  domain  are  applicable  in  this  context  because 
Soldiers  in  these  MOS  have  already  been  selected  through  multiple  hurdles,  which  tend  to 
underestimate  the  relationship  between  predictors  and  criteria  in  the  unrestricted  population. 
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A  number  of  statistically  significant  relationships  are  observed  in  the  25B  MOS.5  ICTL 
scores  are  positively  related  to  perceptions  of  MOS  fit,  which  is  consistent  with  the  notion  of  the 
ICTL  functioning  as  an  indicator  of  interest  and  motivation.  ICTL  scores  are  negatively  related 
to  career  and  reenlistment  intentions  scales,  suggesting  that  more  knowledgeable  Soldiers  are 
less  likely  to  consider  the  Army  as  a  career  or  to  reenlist.  It  should  be  noted  that  such  perceptions 
have  been  measured  at  a  fairly  early  stage  and  Soldiers’  attitudes  at  this  point  are  less  predictive  of 
their  actual  behavior  than  attitudes  captured  at  a  point  more  proximal  to  their  behavior.  That  is, 
Soldiers  are  expressing  attitudes  about  behaviors  or  decisions  that  will  be  made  many  months  or 
years  in  the  future  and  current  attitudes  are  often  weakly  related  to  actual  behaviors  in  the  distant 
future.  ICTL  scores  are  significantly  related  to  WTBD  JKT  scores.  This  relationship  is  likely 
accounted  for  by  the  cognitive  load  of  both  measures.  That  is,  both  the  ICTL  test  and  the  JKT  are  to 
some  extent  indicators  of  general  mental  aptitude,  and  those  with  greater  aptitude  acquire  more 
knowledge  in  both  domains.  ICTL  scores  are  significantly  related  to  the  overall  PRS  mean, 
indicating  that  the  ICTL  test  is  effective  in  discriminating  between  low  and  high  performers  (as 
judged  by  peers)  in  ATT.  ICTL  scores  also  are  significantly  related  to  final  ATT  course  grades,  which 
corroborates  the  former  finding.  Finally,  ICTL  scores  are  significantly  related  to  a  Soldier’s  status  of 
graduating  ATT  without  an  academic  failure.  That  is,  those  with  higher  ICTL  scores  are  more  likely 
to  graduate  ATT  without  an  academic  failure  than  those  with  lower  ICTL  scores. 

ICTL  scores  are  significantly  related  to  perceptions  of  MOS  fit  and  reenlistment  intentions  in 
the  25N  MOS  as  well.  The  directionality  of  these  relationships  is  the  same  as  those  observed  in 
the  25B  MOS,  with  ICTL  scores  positively  related  to  MOS  fit  and  negatively  related  to 
reenlistment  intentions.  While  the  sample  is  markedly  smaller,  ICTL  scores  are  also  significantly 
related  to  WTBD  JKT  scores  and  final  AIT  course  grades  in  the  25N  MOS. 


Table  10 

Bivariate  Correlations  between  ICTL  and  Relevant  Criteria 


25B 


25N 


Criterion  Measure 

n 

r 

P 

p* 

n 

r 

P 

* 

P 

Fit  and  Retention 

Army  Fit 

1.000 

-.015 

.629 

-.038 

152 

-.063 

.438 

oo 
i n 

i 

Attrition  Cognitions 

1.000 

.014 

.665 

.011 

152 

.156 

.056 

.200 

Career  Intentions 

1.000 

-.176 

<.001 

-.248 

152 

-.114 

.161 

-.238 

MOS  Fit 

1.000 

.291 

<.001 

.340 

152 

.160 

.049 

.137 

Reenlistment  Intentions 

1.000 

-.151 

<.001 

-.216 

152 

-.186 

.021 

-.274 

End  of  Training  Job  Knowledge/Performance 

WTBD  %  Correct 

949 

.357 

<.001 

.480 

145 

.177 

.033 

.290 

PRS  Mean 

1.080 

.327 

<.001 

.412 

168 

.145 

.061 

.166 

Final  AIT  Course  Grade 

524 

.405 

<.001 

.492 

159 

.459 

<.001 

.624 

Grad  AIT  w/o  Failure 

1.435 

.158 

<.001 

.215 

228 

.080 

.228 

.156 

*  p  indicates  coefficients  corrected  for  multivariate  range  restriction  on  the  ASVAB  (Lawley,  1943). 
Bold  values  are  statistically  significant  at  the  .01  level.  Italicized  values  are  significant  at  the  .05  level. 


5  Note  that  values  corrected  for  multivariate  range  restriction  are  population  values  to  which  tests  of  statistical 
significance  do  not  apply. 
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Although  correlation  coefficients  are  standard  practice  for  documenting  statistical  evidence 
of  predictive  validity,  interpretation  is  often  fairly  abstract  with  respect  to  practical  implications. 
The  histograms  in  Figures  1-6  present  the  statistically  significant  relationships  observed  in 
Table  10  in  an  expectancy  chart. 

To  create  the  histograms  seen  in  Figures  1-6,  Soldiers  in  each  MOS  were  first  divided  into 
one  of  five  quintiles  based  on  their  standing  on  the  ICTL  test  with  respect  to  the  applicant 
population.  That  is,  the  first  quintile  represents  Soldiers  in  the  bottom  20%  of  the  ICTL  score 
distribution  in  the  applicant  population,  the  second  quintile  represents  Soldiers  with  scores 
falling  between  21%  and  40%  of  the  ICTL  score  distribution  in  the  applicant  population,  and  so 
on  up  to  the  5th  quintile  that  represents  the  top  20%  of  the  ICTL  distribution  in  the  applicant 
population.  Cut  scores  for  the  ICTL  quintiles  were  derived  using  a  separate,  relatively  large 
sample  ( n  =22,829)  of  Army  applicants  administered  at  Military  Entrance  Processing  Stations 
(MEPS).  To  be  clear,  quintiles  were  not  derived  by  dividing  distribution  of  ICTL  scores  in  the 
current  sample  of  25B  and  25N  Soldiers  into  five  rank  ordered  groups  (0-44,  45-49,  50-54,  55- 
59,  and  60-79).  The  current  analysis  sample  of  25B  and  25N  Soldiers  is  made  up  of  Soldiers  who 
have  already  passed  a  number  of  selection  hurdles  (e.g.,  they  have  accessed  into  the  Army  and 
qualified  for  a  selective  MOS)  and  are  therefore  not  representative  of  the  distribution  of  ICTL 
scores  in  the  applicant  population  from  which  the  Army  is  interested  in  selecting  from.  Using  an 
applicant  sample  to  derive  cut  scores  for  the  ICTL  quintiles  is  more  faithful  to  the  selection 
model  this  research  is  ultimately  informing.  One  drawback  of  the  approach  to  using  applicant 
derived  quintile  cuts  is  that  the  five  groups  of  25B  and  25N  Soldiers  are  not  evenly  distributed. 

In  the  most  extreme  example,  there  do  not  happen  to  be  any  25N  Soldiers  with  WTBD  JKT 
scores  in  the  bottom  20%  of  the  applicant  referenced  ICTL  distribution  (see  Ligure  6). 

One  of  the  more  illustrative  relationships  in  Ligures  1-6  includes  the  finding  that  25B 
Soldiers  in  the  top  two  quintiles  have  a  rate  of  graduation  from  AIT  without  a  failure  of  90%, 
compared  to  a  rate  of  70%  for  those  in  the  bottom  quintile  (see  Ligure  4).  Similarly,  25B  Soldiers 
in  the  top  quintile  have  an  average  AIT  final  course  grade  of  86%  compared  to  74%  for  those  in 
the  bottom  quintile  (see  Ligure  4).  Expectancy  charts  are  less  dramatic  in  the  25N  MOS  because 
most  of  those  Soldiers  (75%)  are  in  the  top  two  applicant  referenced  quintiles,  and  the  available 
sample  size  is  relatively  small. 
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Ratine  (1  =  low,  5  =  high)  Rating  (1  =  low,  5  =  high) 


Career  Intentions 

n  =  1,012 

5 


4 


ICTL  Score 


Reenlistment  Intentions 

n  =  1,012 


5 


0-44  45-49  50-54  55-59  60-80 

ICTL  Score 

Figure  1.  Expectancy  charts  for  career  intentions  and  retention  intention  in  the  25  B  MOS. 
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Rating  (1  =  low,  5  =  high) 


MOS  Fit 

n  -  1,012 


5 


4.18 


ICTL  Score 

Figure  2.  Expectancy  charts  for  perception  of  MOS  fit  in  the  25B  MOS. 
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End  Of  Training  Warrior  Tasks  and  Battle  Drill  Job 

KnowledgeTest 

n  =  959 

70% 


66.7% 


ICTL  Score 


Performance  Rating  Scale  Mean 

«  =  2,139 


0-44  45-49  50-54  55-59  60-80 

ICTL  Score 


Figure  3.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance  outcomes  in  the 
25B  MOS. 
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100% 


Advanced  Individual  Training  Final  Course  Grade 

77  =  1,805 


95% 


£  90% 

g  86.3% 


ICTL  Score 

Graduate  Advanced  Individual  Training  without  Failure 

77  =  1,435 

100% 


90.5%  89.9% 


ICTL  Score 

Figure  4.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance  outcomes  in  the 
25B  MOS. 
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Rating  (1  =  low,  5  =  high)  Rating  (1  =  low,  5  =  high) 


Reenlistment  Intentions 

n  =  153 


5 


4.00 


0-44  45-49  50-54  55-59  60-80 

ICTL  Score 


MOS  Fit 

n  =  153 
5 


ICTL  Score 


Figure  5.  Expectancy  charts  for  retention  intention  and  fit  outcomes  of  MOS  fit  in  the  25N  MOS. 
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Average  %  correct  %  correct 


End  of  Training  Warrior  Tasks  and  Battle  Drill  Job  Knowledge  Test 

n  =146 


ICTL  Score 


Advanced  Individual  Training  Final  Course  Grade 

n  =  315 


ICTL  Score 

Figure  6.  Expectancy  charts  for  end  of  training  job  knowledge  and  performance  outcomes  in  the 
25NMOS. 
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Incremental  Validity 

We  examined  incremental  validity  of  the  ICTL  test  over  existing  ASVAB  predictors  by 
testing  a  series  of  hierarchical  regression  models,  regressing  each  criterion  measure  onto 
Soldiers’  ASVAB  based  score  (i.e.,  AFQT,  El  test,  aptitude  area  composite)  in  the  first  step, 
followed  by  their  ICTL  score  in  the  second  step.  The  resulting  increment  in  the  multiple 
correlation  (A R)  when  the  ICTL  score  is  added  to  the  baseline  regression  models  served  as  our 
index  of  incremental  validity.  Lor  the  continuously  scaled  criteria,  the  models  were  estimated 
using  Ordinary  Least  Squares  (OLS)  regression.  Alternatively,  logistic  regression  was  used  for 
the  dichotomous  graduation  criterion  and  the  pseudo  R  value  is  reported  (Nagelkerke,  1991). 

Note  that  although  the  pseudo  R  value  is  intended  to  approximate  the  OLS  R  values,  it  is  not 
directly  comparable  and  should  only  be  used  to  compare  models  within  a  given  nested  set. 

Table  11  presents  the  results  of  incremental  validity  analyses  for  job  knowledge/performance 
criteria  by  MOS.  Ligure  7  presents  much  of  the  same  information  in  the  format  of  a  histogram. 
More  specifically,  each  histogram  presents  the  validity  coefficient  for  the  aptitude  area 
composite  used  for  qualification6  and  the  increment  associated  with  the  ICTL  test.  Incremental 
validity  analyses  have  the  strongest  theoretical  link  to  job  knowledge/performance  criteria 
because  ASVAB  and  ICTL  scores  are  intended  to  predict  task  or  knowledge  based  performance 
outcomes.  The  ICTL  test  provides  appreciable  incremental  validity  beyond  the  ALQT  and  the 
aptitude  area  composites  (i.e.,  EL,  SC,  ST)  when  predicting  AIT  grades  in  both  MOS.  ICTL 
scores  provide  substantial  incremental  validity  beyond  the  El  test  in  predicting  AIT  grades  and 
WTBD  JKT  scores  in  both  MOS.  With  regard  to  the  25B  MOS  only,  the  ICTL  test  provides 
statistically  significant  and  practically  meaningful  incremental  validity  beyond  all  ASVAB-based 
composites  evaluated  when  predicting  PRS  means  and  graduation  from  AIT  without  a  failure. 

Table  12  presents  the  results  of  incremental  validity  analyses  for  fit  and  retention  related 
criteria  by  MOS.  Ligure  8  presents  much  of  the  same  information  in  the  format  of  a  histogram. 
Relatively  fewer  statistically  significant  results  are  observed.  Moreover,  a  number  of  the 
relationships  are  negative.  Note  that  multiple  correlation  values  presented  in  Table  12  reflect 
only  the  strength  of  relationship  and  not  the  direction  (i.e.,  R  cannot  achieve  negative  values). 

The  most  interesting  finding  is  that  the  ICTL  test  provides  the  greatest  incremental  validity 
beyond  the  ASVAB  in  predicting  perceptions  of  MOS  fit  in  the  25B  MOS,  and  that  this 
relationship  is  a  positive  one.  ICTL  scores  also  provide  incremental  validity  beyond  the  aptitude 
area  composites  in  predicting  MOS  fit  in  the  25N  MOS.  It  is  likely  that  the  ICTL  test  captures 
unique,  job  specific  variance  in  this  relationship  that  cannot  be  accounted  for  by  the  general 
aptitude  variance  component  it  shares  with  the  ASVAB  based  predictors.  That  is,  both  the 
ASVAB  and  ICTL  tests  capture  general  aptitude  and  it  may  be  that  those  of  higher  general 
aptitude  perceive  better  MOS  fit  because  they  have  a  higher  degree  of  success  in  a  challenging 
MOS.  The  ICTL  test  also  captures  unique  variance  that  is  conceptually  distinct  from  general 
aptitude  and  specifically  related  to  the  25B  and  25N  MOS.  This  conceptual  link  between  the 
content  of  the  ICTL  and  the  nature  of  the  MOS  may  be  a  reflection  of  the  “information  test” 


6  Note  that  25N  requires  qualification  on  both  Electronics  (EL)  and  Surveillance  and  Communications  (SC)  aptitude 
area  composites.  These  two  composite  scores  have  a  correlation  of  .99  in  the  current  sample,  so  the  figures  only 
present  increment  over  the  SC  composite. 
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notion  discussed  earlier  that  postulates  individuals  gravitate  toward  knowledge  areas  for  which 
they  have  interest  and  motivation. 

The  general  trend  observed  in  Table  12  among  the  other  criterion  variables  suggests  those  of 
higher  aptitude  tend  to  be  less  intent  on  reenlisting  or  making  the  Army  a  career.  The  ICTL  test 
provides  a  small  to  moderate  amount  of  incremental  validity  beyond  ASVAB  based  measures  in 
this  regard. 

Tables  13  and  14  present  the  incremental  validity  analyses  corrected  for  multivariate  range 
restriction  on  the  ASVAB  (Lawley,  1943).  As  with  the  bivariate  relationships  discussed  above, 
we  use  a  large  sample  ( n  =  483,737)  of  Army  applicants  as  the  unrestricted  reference  (see  Knapp 
&  LaPort,  2014  for  details  of  the  sample).  In  general,  the  pattern  of  relationships  and  amount  of 
incremental  variance  provided  is  comparable  to  what  is  found  in  the  restricted  analyses  found  in 
Tables  1 1  and  12.  The  correction  for  multivariate  range  restriction  tended  to  result  in  a  slightly 
larger  adjustment  for  the  ASVAB  test  and  composites  than  for  the  ICTL  test.  This  is  likely 
because  the  variances  of  ASVAB  test  and  composites  are  adjusted  directly  to  the  population 
values  and  the  ICTL  test  variances  are  indirectly  adjusted  based  on  their  covariance  with  the 
ASVAB. 
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Table  11 


Incremental  Validity  ofICTL  in  Predicting  Job  Knowledge/Performance  Criteria 


AFQT+ 

EI+ 

EL  + 

SC  + 

ST  + 

AFQT  r  ICTL  R 

A  R 

El  r  ICTL  R  A  R 

EL  r 

ICTL  R  A  R 

SC  r  ICTL  R  A  R 

STr 

ICTL  R  A  R 

25  B 


WTBD  %  Correct 

.44 

.46 

.02 

.32 

.39 

.07 

.44 

.45 

.01 

.44 

.45 

.01 

.45 

.46 

.01 

PRS  Mean 

.30 

.36 

.06 

.22 

.33 

.11 

.33 

.37 

.04 

.33 

.37 

.04 

.33 

.37 

.04 

Final  AIT  Course  Grade 

.39 

.45 

.06 

.31 

.42 

.11 

.45 

.48 

.03 

.46 

.49 

.03 

.46 

.48 

.03 

Grad  AIT  w/o  Failure* 

.22 

.25 

.03 

.06 

.22 

.15 

.22 

.24 

.03 

.23 

.25 

.02 

.22 

.25 

.02 

251' 

WTBD  %  Correct 

.18 

.23 

.05 

.05 

.20 

.15 

.13 

.20 

.07 

.12 

.20 

.08 

.13 

.20 

.07 

PRS  Mean 

.04 

.12 

.09 

.12 

.15 

.03 

.09 

.13 

.04 

.09 

.13 

.04 

.08 

.13 

.05 

Final  AIT  Course  Grade 

.45 

.54 

.07 

.26 

.47 

.19 

.45 

.52 

.07 

.47 

.53 

.06 

.47 

.53 

.06 

Grad  AIT  w/o  Failure* 

.14 

.15 

.02 

.06 

.12 

.06 

.11 

.14 

.03 

.13 

.15 

.02 

.13 

.15 

.02 

Note :  AFQT  =  Armed  Forces  Qualification  Test;  El  =  Electronics  Information  test;  EL  =  Electronics  composite;  SC  =  Surveillance  and  Communication  composite;  ST  =  Skilled 
Technical  composite. 

Bold  values  are  statistically  significant  at  the  .01  level.  Italicized  values  are  significant  at  the  .05  level. 

*Dichotomous  criterion — pseudo  R  reported.  Compare  only  within  a  row. 

Table  12 

Incremented  Vcdidity  ofICTL  in  Predicting  Fit  and  Retention  Criteria 


AFQT 

r 

AFQT+ 
ICTL  R 

A  R 

El  r 

EI+ 

ICTL  R 

A  R 

ELr 

EL  + 
ICTL  R 

A  R 

SCr 

SC  + 
ICTL  R 

A  R 

STr 

ST  + 
ICTL  R 

A  R 

25  E 

» 

Army  Fit 

-.04 

.04 

.00 

-.02 

.02 

.00 

-.05 

.05 

.00 

-.05 

.05 

.00 

-.05 

.05 

.00 

Attrition  Cognitions 

-.01 

.02 

.01 

-.02 

.03 

.01 

-.01 

.03 

.02 

-.01 

.03 

.02 

-.01 

.03 

.02 

Career  Intentions 

-.17 

.20 

.03 

-.09 

.18 

.09 

-.18 

.20 

.02 

-.18 

.20 

.02 

-.19 

.20 

.02 

MOS  Fit 

.20 

.30 

.10 

.22 

.30 

.08 

.23 

.30 

.07 

.23 

.30 

.07 

.22 

.30 

.08 

Reenlistment  Intentions 

-.16 

.18 

.02 

-.08 

.15 

.07 

-.16 

.17 

.01 

-.16 

.17 

.01 

-.17 

.18 

.01 

2515 

1 

Army  Fit 

.00 

.12 

.12 

-.16 

.17 

.01 

-.14 

.15 

.01 

-.12 

.14 

.02 

-.11 

.13 

.02 

Attrition  Cognitions 

-.01 

.19 

.18 

.16 

.20 

.04 

.12 

.18 

.06 

.10 

.18 

.08 

.09 

.18 

.09 

Career  Intentions 

-.23 

.23 

.00 

-.06 

.10 

.04 

-.21 

.21 

.00 

-.20 

.20 

.00 

-.21 

.21 

.00 

MOS  Fit 

.02 

.16 

.14 

.07 

.16 

.09 

-.02 

.19 

.17 

-.01 

.18 

.17 

-.01 

.18 

.17 

Reenlistment  Intentions 

-.17 

.20 

.03 

-.10 

.17 

.07 

-.20 

.21 

.01 

-.19 

.21 

.02 

-.19 

.21 

.02 

Note'.  AFQT  =  Armed  Forces  Qualification  Test;  El  =  Electronics  Information  test;  EL  =  Electronics  composite;  SC  =  Surveillance  and  Communication  composite;  ST  =  Skilled 
Technical  composite. 

Bold  values  are  statistically  significant  at  the  .01  level.  Italicized  values  are  significant  at  the  .05  level. 

*Dichotomous  criterion — pseudo  R  reported.  Compare  only  within  a  row. 
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Table  13 

Incremental  Validity  ofICTL  in  Predicting  Job  Knowledge/Performance  Criteria  Corrected  for  Multivariate  Range  Restriction  on  the 
ASVAB 


AFQT  p 

AFQT+ 

ICTLP 

A  P 

El  p 

EI+ 

ICTL  P 

A  P 

EL 

P 

EL  + 
ICTL  P 

A  P 

SCp 

SC  + 
ICTLP 

A  P 

ST  p 

ST  + 
ICTL  P 

A  P 

25  E 

1 

WTBD  %  Correct 

.56 

.58 

.02 

.43 

.51 

.08 

.56 

.57 

.01 

.56 

.57 

.01 

.57 

.58 

.01 

PRS  Mean 

.41 

.45 

.04 

.31 

.42 

.11 

.43 

.45 

.02 

.43 

.46 

.02 

.43 

.46 

.02 

Final  AIT  Course  Grade 

.51 

.55 

.04 

.39 

.51 

.12 

.54 

.56 

.02 

.56 

.57 

.02 

.55 

.57 

.02 

251' 

1 

WTBD  %  Correct 

.29 

.33 

.04 

.19 

.29 

.10 

.26 

.31 

.05 

.26 

.31 

.05 

.26 

.31 

.05 

PRS  Mean 

.12 

.17 

.05 

.18 

.20 

.02 

.17 

.19 

.02 

.17 

.19 

.02 

.16 

.18 

.02 

Final  AIT  Course  Grade 

.68 

.72 

.04 

.49 

.64 

.15 

.68 

.71 

.03 

.69 

.72 

.03 

.69 

.72 

.03 

Note :  AFQT  =  Armed  Forces  Qualification  Test;  El  =  Electronics  Information  test;  EL  =  Electronics  composite;  SC  =  Surveillance  and  Communication  composite;  ST  =  Skilled 
Technical  composite. 

p  indicates  coefficients  that  were  corrected  for  multivariate  range  restriction  (Lawley,  1943). 


Table  14 

Incremental  Validity  ofICTL  in  Predicting  Fit  and  Retention  Criteria  Corrected  for  Multivariate  Range  Restriction  on  the  ASVAB 


AFQT  p 

AFQT+ 

ICTLP 

A  P 

El  p 

EI+ 

ICTL  P 

A  P 

EL 

P 

EL  + 
ICTL  P 

A  P 

SCp 

SC  + 
ICTLP 

A  P 

ST  p 

ST  + 
ICTLP 

A  P 

25  E 

\ 

Army  Fit 

-.06 

.06 

.00 

-.04 

.04 

.00 

-.07 

.07 

.00 

-.07 

-.07 

.00 

-.07 

.07 

.00 

Attrition  Cognitions 

-.01 

.03 

.02 

-.01 

.03 

.01 

-.01 

.02 

.01 

-.01 

.02 

.01 

-.01 

.03 

.02 

Career  Intentions 

-.25 

.27 

.02 

-.18 

.25 

.07 

-.26 

.28 

.01 

-.26 

.27 

.01 

-.26 

.28 

.01 

MOS  Fit 

.27 

.34 

.07 

.27 

.35 

.08 

.29 

.35 

.06 

.29 

.35 

.06 

.29 

.34 

.06 

Reenlistment  Intentions 

-.22 

.24 

.02 

-.16 

.22 

.06 

-.23 

.24 

.01 

-.23 

.24 

.01 

-.24 

.25 

.01 

25I'> 

1 

Army  Fit 

-.07 

.16 

.09 

-.24 

.24 

.00 

-.20 

.20 

.00 

-.18 

.19 

.01 

-.17 

.18 

.01 

Attrition  Cognitions 

.04 

.22 

.18 

.22 

.24 

.02 

.16 

.20 

.04 

.14 

.20 

.06 

.13 

.20 

.07 

Career  Intentions 

-.36 

.36 

.00 

-.28 

.29 

.01 

-.37 

.37 

.00 

-.37 

.37 

.00 

-.36 

.36 

.00 

MOS  Fit 

.01 

.16 

.15 

.04 

.15 

.11 

-.03 

.20 

.17 

-.02 

.19 

.17 

-.02 

.19 

.17 

Reenlistment  Intentions 

-.31 

.33 

.02 

-.27 

.31 

.04 

-.34 

.35 

.01 

-.34 

.35 

.01 

-.33 

.34 

.01 

Note'.  AFQT  =  Armed  Forces  Qualification  Test;  El  =  Electronics  Information  test;  EL  =  Electronics  composite;  SC  =  Surveillance  and  Communication  composite;  ST  =  Skilled 
Technical  composite. 

p  indicates  coefficients  that  were  corrected  for  multivariate  range  restriction  (Lawley,  1943). 
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25B 


Final  AIT 
Course  Grade 


Grad  AIT  w/o 
Failure* 


0  0.2  0.4 

Prediction  Potential 


■  Skilled  Technical 
composite 

=  ICTL 


0.6 


25N 


WTBD  % 
Correct 


PRS  Mean 


Final  AIT 
Course  Grade 


■  Surveillance  and 
Communication 
composite 

=  ICTL 


Grad  AIT  w/o 
Failure* 

| - 1 - 1 - 1 - 1 - 1 - 1 - 1 - - 1 - 1 - 1 - 1 - 1 - 1 - 1 - - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 

0  0.2  0.4  0.6 

Prediction  Potential 


Figure  7.  Incremental  validity  of  ICTL  in  predicting  job  knowledge/performance  criteria. 
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25B 


Army  Fit 

Attrition 

Cognitions 

Career 

Intentions 

MOS  Fit 


Reenlistment 

Intentions 


0 


Army  Fit 


Attrition 

Cognitions 

Career 

Intentions 


MOS  Fit 


Reenlistment 

Intentions 


■  Skilled  Technical 
composite 
=  ICTL 


0.2  0.4 

Prediction  Potential 

25N 


0.6 


n - 1 - 1 - 1 - r~ 


■  Surveillance  and 
Communication 
composite 
=  ICTL 


“i - 1 - 1 - r~ 


0  0.2  .  0.4  .  0.6 

Prediction  Potential 

Figure  8.  Incremental  validity  of  ICTL  in  predicting  job  knowledge/performance  criteria.1 


7  The  absence  of  a  patterned  bar  in  the  histogram  indicates  that  no  evidence  of  incremental  validity  was  observed. 
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Fairness  Analyses 

As  a  final  component  to  the  project,  we  examined  group  differences  in  test  scores  as  well  as 
predictive  bias  with  regard  the  ICTL  test  for  gender  and  racial/ethnic  subgroups.  Subgroup 
differences  and  disparities  in  pass  rates  are  important  to  monitor  as  they  reflect  how  use  of  a  test 
or  selection  system  impacts  the  demographic  composition  of  incoming  Soldiers  and  the  extent  to 
which  that  composition  is  representative  of  the  population  at  large.  Furthermore,  from  an  ethical 
and  fairness  standpoint,  predictors  should  not  be  biased  against  groups  due  to  factors  unrelated  to 
performance  requirements  as  a  Soldier. 

In  this  section,  we  first  report  information  pertaining  to  standardized  subgroup  mean 
differences  on  the  ICTL  test  for  gender  and  racial/ethnic  subgroups  in  both  25B  and  25N  MOS. 
Next,  we  examine  the  presence  and  extent  of  differential  validity,  pertaining  to  group  differences 
in  validity  coefficients,  and  differential  prediction,  pertaining  to  group  differences  in  the 
regression  of  criterion  outcomes  on  predictor  scores  (Berry,  Clark,  &  McClure,  2011). 
Differential  validity  and  prediction  analyses  focused  on  AIT  grades,  AIT  graduation  status,  and 
peer  performance  rating  scales  (PRS).  In  addition,  because  the  ICTL  test  can  be  thought  of  as  an 
indirect  measure  of  interest,  intrinsic  motivation,  and  skill  relevant  to  cyber-related  work,  we 
also  included  MOS  fit  as  outcome  for  differential  validity  and  prediction  analyses.  For  analyses 
reported  in  this  subsection,  five  groups  were  examined:  males  (25B  n  =  1,371;  25N  n  =  254), 
females  (25B  n  =  359;  25N  n  =  39),  non-Hispanic  Blacks  (25B  n  =  522;  25N  n  =  62),  non- 
Hispanic  Whites  (25B  n  =  803;  25N  n  =  164),  and  Hispanic  Whites  (25B  n  =  247;  25N  n  =  37). 
Analyses  were  carried  out  for  three  comparisons:  male  vs.  female,  non-Hispanic  White  vs. 
Hispanic  White,  and  non-Hispanic  White  vs.  non-Hispanic  Black.  These  groups  were  chosen  to 
be  consistent  with  designations  used  by  the  ASVAB  testing  program  (Defense  Manpower  Data 
Center,  2011). 

Given  common  levels  of  validity,  criterion  reliability,  and  other  contextual  factors  affecting 
statistical  power,  differential  validity  and  differential  prediction  analyses  require  relatively  large 
overall  and  subgroup  sample  sizes  in  order  to  be  sufficiently  powerful  (e.g.,  Aguinis,  Culpepper, 
&  Pierce,  2010);  consequently,  these  analyses  were  conducted  only  for  25B  given  the  relatively 
small  minority  group  sample  sizes  for  25N.  Finally,  to  provide  a  frame  of  reference  for  the 
results  observed  for  the  ICTL  test,  we  also  examine  results  for  the  ASVAB  Electronics 
Information  (El)  subtest  and  several  area  aptitude  composites  relevant  to  cyber  occupations  in 
the  Armed  Forces,  namely  the  Skilled-Technical  (ST),  Electrical  (EL),  and  Surveillance  and 
Communication  (SC)  Composites. 

Results  pertaining  to  the  fairness  analyses  are  shown  in  Tables  15  and  16  (more  detailed 
results  are  available  in  the  Appendix;  see  Tables  A6-A14).  Table  15  shows  results  pertaining  to 
subgroup  mean  differences  for  the  ICTL  test  and  ASVAB  subtest  and  composites  within  the  25B 
and  25N  MOS.  For  25B,  d  values  for  the  ICTL  test  were  in  the  0.40-0.60  SD  range  across  the 
comparisons  conducted;  these  differences  would  be  considered  relatively  moderate  in  magnitude 
(Cohen,  1992).  For  each  comparison  within  25B,  group  mean  differences  for  the  ICTL  test  were 
uniformly  smaller  than  corresponding  differences  for  the  ASVAB  scores.  For  25N,  d  values  for 
the  ICTL  test  were  somewhat  smaller  than  those  observed  in  for  25B,  ranging  from 
approximately  -.20  to  -0.50  across  the  comparisons  conducted.  With  one  exception  (the  El 
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subtest  for  the  Black/Whitc  comparison),  group  mean  differences  for  the  ICTL  test  were 
uniformly  smaller  than  those  observed  for  the  ASVAB  El  subtest  and  area  composites,  a  finding 
which  is  similar  to  that  for  25B. 

Table  16  shows  results  pertaining  to  differential  validity  and  differential  prediction  analyses 
for  Female/Male,  Hispanic/White,  and  Black/White  comparisons.  Berry,  Cullen  and  Meyer 
(2014)  found  that  subgroup  differences  in  range  restriction  in  military  samples,  arising  from 
strong  reliance  on  cognitive  tests  in  conjunction  with  strict  use  of  cutoff  scores,  may  act  as  a 
partial  source  of  observed  group  differences  in  criterion-related  validity  estimates  (see  also  Roth, 
Le,  Oh,  van  Iddekinge,  Buster,  Robbins,  &  Campion,  2014).  Consequently,  we  corrected  for 
multivariate  range  restriction  (Lawley,  1943)  using  each  subgroup’s  covariance  matrix  and 
conducted  subsequent  analyses  using  the  within-group  corrected  matrices.  Significance  tests 
were  carried  out  only  on  the  uncorrected  estimates  in  Table  16  due  to  lack  of  analytical  standard 
errors  and  test  statistics  for  the  corrected  estimates. 

For  the  ICTF  test,  only  one  instance  of  a  non-zero  between-group  difference  in  validity  was 
observed  for  the  White/Hispanic  comparison  (MOS  fit).  Multiple  instances  of  group  differences 
in  validity  were  observed  for  the  El  subtest  and  composites  for  each  comparison,  including 
gender  (AIT  grades,  PRS),  White/Hispanic  (AIT  grades,  AIT  graduation  status,  and  MOS  fit), 
and  White/Black  (PRS). 

With  respect  to  differential  prediction  (indicated  by  a  significant  predictor-group  interaction 
term  in  the  regression  of  an  outcome  on  the  predictor,  group  membership,  and  the 
aforementioned  interaction  term),  there  were  no  observed  instances  for  the  ICTF  test  where  the 
difference  suggested  a  steeper  slope  for  the  majority  group,  as  indicated  by  a  negative  estimate. 
Rather,  the  one  instance  of  a  significant  predictor-group  interaction  term  (Black/White 
comparison  for  AIT  graduation  status)  indicated  that  the  minority  group  had  a  slightly  higher 
slope  than  did  the  majority  group  meaning  the  predictor  was  more  predictive  for  the  comparison 
group.  In  multiple  instances,  significant  predictor- group  interactions  were  found  involving  the 
ASVAB  El  subtest  or  area  composites.  For  gender,  significant  interactions  were  found  involving 
the  El  subtest  in  predicting  both  AIT  grades  and  PRS.  For  the  Hispanic/White  comparison, 
significant  interactions  were  found  involving  the  El  subtest  in  predicting  AIT  grades  and  for  the 
El  subtests  and  all  area  composites  in  predicting  MOS  fit.  Aside  from  the  aforementioned 
interaction  involving  the  El  subtest  for  AIT  graduation  status,  no  additional  predictor-group 
interactions  were  found  for  the  Black/White  comparison. 
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Table  15 

Gender  and  Racial/Ethnic  Subgroup  Standardized  Group  Mean  Differences  (Cohen ’s  d)  among  Predictor  and  Criterion  Variables  in 
25 B  and  25N  MOS 


25  B 

25N 

Measure 

Female  / 

Male 

Black  / 

White 

Hispanic  / 
White 

Female  / 

Male 

Black  / 

White 

Hispanic  / 
White 

Predictors 

ICTL  Scaled  Score 

-0.52 

-0.62 

-0.42 

-0.34 

-0.53 

-0.19 

ASVAB:  El  Subtest 

-0.84 

-0.77 

-0.49 

-0.70 

-0.44 

-0.41 

ASVAB:  Skilled-Technical  Composite 

-0.62 

-0.91 

-0.67 

-0.44 

-0.77 

-0.61 

ASVAB:  Electrical  Composite 

-0.72 

-0.96 

-0.67 

-0.56 

-0.79 

-0.64 

ASVAB:  Surveillance  and  Communication 

-0.65 

-0.89 

-0.64 

-0.47 

-0.72 

-0.57 

Composite 

Criteria 

Final  AIT  course  grade 

-0.10 

-0.46 

-0.18 

-0.23 

-0.45 

0.16 

AIT  graduation  status 

0.05 

-0.16 

-0.04 

0.23 

0.11 

-0.04 

Peer  performance  rating  scales  (PRS) 

-0.32 

-0.30 

-0.23 

-0.19 

0.18 

-0.01 

MOS  fit 

-0.57 

-0.21 

-0.17 

-0.64 

0.65 

-0.14 

Note.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  subgroup  sample  size  is  1,371  for  Males,  359  for 
Females,  522  for  Blacks,  247  for  Hispanics,  and  803  for  Whites.  For  25N,  subgroup  sample  size  is  254  for  Males,  39  for  Females, 
62  for  Blacks,  37  for  Hispanics,  and  164  for  Whites.  Significant  Cohen's  d  values,  based  on  an  independent  sample  r-test  between 
the  group  means,  are  bolded  ( p  <  .05,  two-tailed). 
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Table  16 

Gender  and  Racial/Ethnic  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for  Differential 
Prediction  Analyses 

Female/Male  Hispanic/White  Black/White 


Correlations  Predictor  x  Group 

Correlations  Predictor  x  Group 

Correlations 

Predictor  x  Group 

Male  Female 

White  Hispanic 

White  Black 

Criterion  /  Predictor 

r 

f  RRC 

r 

f  RRC 

b 

b  RRC 

r 

f  RRC 

r 

r  RRC 

b 

b  RRC 

r 

f  RRC 

r 

f  RRC 

b 

b  RRC 

Final  AIT  course  grade 

ICTL  Scaled  Score 

.43 

.50 

.28 

.43 

-0.12 

-0.01 

.41 

.46 

.29 

.36 

-0.17 

-0.17 

.41 

.46 

.32 

.46 

0.06 

0.24 

ASVAB:  El  Subtest 

.36 

.41 

.10 

.31 

-0.28 

-0.08 

.34 

.37 

.08 

.17 

-0.30 

-0.27 

.34 

.37 

.22 

.39 

-0.05 

0.14 

ASVAB:  Skilled-Technical  Composite 

.49 

.56 

.28 

.49 

-0.08 

-0.03 

.52 

.57 

.28 

.35 

-0.18 

-0.17 

.52 

.57 

.37 

.56 

0.12 

0.15 

ASVAB:  Electrical  Composite 

.49 

.56 

.29 

.48 

-0.07 

-0.02 

.51 

.56 

.26 

.34 

-0.17 

-0.17 

.51 

.56 

.36 

.56 

0.11 

0.15 

ASVAB:  Surveillance  and  Communication 
Composite 

.50 

.57 

.29 

.49 

-0.08 

-0.03 

.52 

.57 

.27 

.35 

-0.17 

-0.17 

.52 

.57 

.39 

.57 

0.15 

0.16 

AIT  graduation  status 

ICTL  Scaled  Score 

.17 

.23 

.13 

.23 

().(X) 

0.00 

.14 

.17 

.10 

.09 

().(X) 

0.00 

.14 

.17 

.21 

.30 

0.01 

0.01 

ASVAB:  EI  Subtest 

.07 

.13 

.01 

.11 

().(X) 

0.00 

.08 

.09 

-.02 

.00 

().(X) 

0.00 

.08 

.09 

.00 

.11 

0.00 

0.00 

ASVAB:  Skilled-Technical  Composite 

.17 

.24 

.14 

.27 

0.00 

0.00 

.19 

.22 

.03 

.04 

0.00 

0.00 

.19 

.22 

.14 

.28 

0.00 

0.00 

ASVAB:  Electrical  Composite 

.17 

.23 

.13 

.26 

().(X) 

0.00 

.18 

.21 

.03 

.04 

().(X) 

0.00 

.18 

.21 

.14 

.28 

0.00 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

.18 

.24 

.14 

.27 

().(X) 

0.00 

.19 

.22 

.04 

.04 

().(X) 

0.00 

.19 

.22 

.15 

.29 

0.00 

0.00 

Peer  performance  rating  scales  (PRS) 

ICTL  Scaled  Score 

.33 

.41 

.23 

.30 

().(X) 

-0.01 

.34 

.40 

.26 

.31 

().(X) 

0.00 

.34 

.40 

.23 

.33 

-0.01 

0.00 

ASVAB:  EI  Subtest 

.22 

.31 

.01 

.13 

-0.02 

-0.01 

.24 

.27 

.12 

.19 

-0.01 

-0.01 

.24 

.27 

.10 

.21 

-0.01 

0.00 

ASVAB:  Skilled-Technical  Composite 

.34 

.44 

.13 

.25 

-0.01 

-0.01 

.36 

.42 

.22 

.27 

().(X) 

-0.01 

.36 

.42 

.21 

.35 

0.00 

0.00 

ASVAB:  Electrical  Composite 

.34 

.43 

.12 

.25 

-0.01 

-0.01 

.35 

.41 

.24 

.29 

0.00 

-0.01 

.35 

.41 

.20 

.34 

0.00 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

.34 

.44 

.12 

.24 

-0.01 

-0.01 

.35 

.42 

.23 

.28 

().(X) 

-0.01 

.35 

.42 

.20 

.35 

0.00 

0.00 

MOS  fit 

ICTL  Scaled  Score 

.26 

.30 

.23 

.34 

0.00 

0.01 

.34 

.37 

.15 

.16 

-0.02 

-0.02 

.34 

.37 

.21 

.27 

-0.01 

-0.01 

ASVAB:  EI  Subtest 

.16 

.20 

.18 

.29 

0.01 

0.01 

.26 

.29 

.06 

.07 

-0.02 

-0.02 

.26 

.29 

.14 

.19 

-0.01 

-0.01 

ASVAB:  Skilled-Technical  Composite 

.18 

.24 

.17 

.32 

0.01 

0.01 

.27 

.31 

.02 

.07 

-0.02 

-0.01 

.27 

.31 

.14 

.23 

-0.01 

0.00 

ASVAB:  Electrical  Composite 

.18 

.23 

.20 

.33 

0.01 

0.01 

.27 

.31 

.02 

.06 

-0.02 

-0.01 

.27 

.31 

.15 

.24 

-0.01 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

.18 

.24 

.19 

.33 

0.01 

0.01 

.26 

.31 

.02 

.06 

-0.02 

-0.01 

.26 

.31 

.16 

.24 

0.00 

0.00 

Note  .  Uncorrected  estimates  based  on  the  validation  sample;  corrected  estimates  based  on  range  restriction  corrections  as  explained  in  text.  The  values  for  the  Predictor  xGroup 
cross-product  terms  are  unstandardized  coefficients  for  a  model  that  includes  the  focal  predictor  measure,  demographic  variable  (coded  as  0  =  Majority,  1  =  Minority),  and  the 
interaction  between  the  focal  predictor  and  the  demographic  variable.  Sample  sizes  for  Males  range  ffom413  to  1,132;  for  Females,  from  111  to  303;  for  Whites,  from245  to  642;  for 
Hispanics,  from71  to  219,  and;  for  Blacks,  from  157  to  442.  For  all  tabled  values,  bolded  values  are  those  where  p  <  .05.  Italicized  correlations  are  those  where  the  difference  between 
the  Majority  and  Minority  estimates  is  significant  at  two-tailed  p  <  .05. 
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Discussion 


There  are  a  few  issues  to  keep  in  mind  in  evaluating  these  results.  First,  incremental  validity 
analyses  do  not  reflect  the  way  the  ASVAB  is  used  operationally.  Incremental  validity  analyses 
assume  that  the  ASVAB  is  used  optimally  (i.e.,  as  a  top-down  selection  tool).  The  assumption  is 
not  valid.  Each  Service  sets  a  minimum  AFQT  score  for  entry  into  the  Service;  while  AFQT  is  a 
continuous  variable,  it  is  used  dichotomously.  Aptitude  area  composites  are  used  in  a  similar 
manner  for  MOS  assignment.  The  question  incremental  validity  analyses  really  address  is  “how 
much  additional  prediction  would  the  new  test  provide,  if  the  ASVAB  were  used  optimally.”  As 
the  ASVAB  is  not  used  optimally,  these  incremental  validity  estimates  are  conservative  and  may 
underestimate  the  actual  selection  efficiency  of  the  new  test. 

Regardless,  incremental  validity  is  an  index  the  Services  have  been  using  to  evaluate  new 
predictors  for  more  than  20  years.  Importantly,  the  estimates  reported  here  are  comparable  to 
those  previously  reported  for  the  science  and  technical  tests  of  the  ASVAB.  For  example, 

Oppler,  Russell,  Rosse,  Keil,  Meiman,  and  Welsh  (1997)  reported  incremental  validities  from  a 
Joint-Service  study  that  included  13  technical  training  schools.  Validities  for  each  ASVAB  test 
were  computed  using  only  the  training  schools  that  included  that  test  in  their  selection 
composites.  For  example,  10  training  schools  used  General  Science  in  their  composites  and  only 
those  10  schools  were  included  in  the  validity  estimates.  Average  incremental  validity  estimates 
beyond  AFQT,  after  correction  for  multivariate  range  restriction,  ranged  from  .012  for  El  to  .034 
for  Auto  and  Shop  Information.  It  is  also  important  to  note  that  military  research  suggests  that 
even  small  amounts  of  incremental  validity  (e.g.,  .02)  can  have  utility  in  large  selection  programs 
(Held,  Fedak,  Crookenden,  &  Blanco,  2002;  Schmidt,  Dunn,  &  Hunter,  1995). 

Summary  and  Conclusion 

Results  suggest  that  the  ICTF  test  has  potential  as  a  valid  and  highly  efficient  predictor  of 
valued  outcomes  in  United  States  Army  Cyber  Center  of  Excellence  MOS.  Not  only  is  the  ICTF 
test  a  valid  predictor  of  job  knowledge  and  performance  related  criteria  such  as  course  grades, 
but  is  also  a  valid  predictor  of  perceived  MOS  fit.  This  finding  lends  support  to  the  notion  of  the 
ICTF  test  functioning  as  an  indirect  measure  of  interest,  intrinsic  motivation,  and  skill  in  a 
particular  area.  Just  as  the  Automotive  and  Shop  (AS)  test  can  be  thought  of  as  a  way  to  identify 
hobbyists  who  like  to  work  on  cars  or  motorcycles  and  are  therefore  more  likely  to  perceive 
better  fit  in  automotive  related  MOS  (e.g.,  Fight  Wheel  Vehicle  Mechanic;  91B),  the  ICTF  is 
likely  operating  at  some  level  to  capture  variance  related  to  applicants  in  the  information 
technology  (IT)  domain  who  like  to  do  things  like  build  computers  and  configure  elaborate  home 
networks. 

What  is  perhaps  most  notable  about  the  pattern  of  validity  and  incremental  validity  results 
found  in  Tables  11  through  14  is  the  ICTF  test’s  efficiency  of  prediction  in  these  Signal  MOS.  In 
general,  the  ICTF  test  predicts  performance  just  as  well  as  composites  derived  from  multiple 
ASVAB  tests.  Moreover,  the  ICTF  test  explains  additional  variance  beyond  these  composites  in 
almost  every  criterion  measure.  Validity  of  the  ICTF  test  is  substantially  greater  than  its  closest 
counterpart  in  the  ASVAB,  the  El  test,  in  predicting  performance  in  these  particular  MOS.  Thus 
it  represents  a  useful  supplement  to  ASVAB  for  cyber  occupations. 
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Appendix 


Table  A1 

25B  Training  Performance  Rating  Scales 

Learns  to  implement  a  network 

•  Identifies  and  installs  network  infrastructure  devices 

•  Develops  and  manages  an  IP  address  scheme. 

•  Integrates  network  topology  design. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your 
class,  but  not  as 
good  as  the  top 
40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 
Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  hardware  concepts 

•  Identifies  hardware  components. 

•  Recognizes  hardware  compatibility. 

•  Installs,  replaces,  and  maintains  hardware. 

1  I  2  I  3 

Among  the  Weakest  Below  Average  (in  Average  (better  than 
(in  the  bottom  20%  the  bottom  40%  of  the  bottom  40%  of 
of  Soldiers  in  your  Soldiers  in  your  Soldiers  in  your  class, 

class)  class)  but  not  as  good  as  the 

top  40%) 

Learns  to  administer  software  applications 

Installs  operating  systems  (e.g.,  Windows,  Linux). 

Administers  software  changes  using  command  line  and  software  utilities. 
Recognizes  software  compatibility. 

Performs  basic  software  operations  on  DoD-approved  software. 

~T~  I  2  I  3  ~ 

Among  the  Weakest  Below  Average  (in  Average  (better  than  the 

(in  the  bottom  20%  the  bottom  40%  of  bottom  40%  of  Soldiers 

of  Soldiers  in  your  Soldiers  in  your  in  your  class,  but  not  as 

class)  class)  good  as  the  top  40%) 

Learns  the  fundamentals  of  network  security 

•  Implements  security  policies  (e.g.,  password  security). 

•  Controls  access  to  network  resources. 

•  Identifies  different  classification  levels  (e.g.,  green  vs.  blue  vs.  red). 

•  Performs  incident  handling  procedures. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average 
(in  the  bottom 
40%  of  Soldiers  in 
your  class) 

Average  (better  than  the 
bottom  40%  of  Soldiers 
in  your  class,  but  not  as 
good  as  the  top  40%) 

Above  Average 
(in  the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

4 _ 5 _ 

Above  Average  Among  the 
(in  the  top  40%  Strongest  (in  the  top 
of  Soldiers  in  20%  of  Soldiers  in 

your  class)  your  class) 


4 _ 5 _ 

Above  Average  Among  the 

(in  the  top  40%  Strongest  (in  the  top 
of  Soldiers  in  20%  of  Soldiers  in 
your  class)  your  class) 


A-l 


Table  A1  (continued) 

25B  Training  Performance  Rating  Scales 

Learns  the  basics  of  troubleshooting 

Learns  how  signals  flow  through  equipment. 

Explains  solutions  to  users. 

Consults  flowcharts,  diagrams,  and  other  resources  (e.g.,  Microsoft  Knowledge  Base). 
Inspects  and  services  hardware  and  software  (e.g.,  preventative  checks  and  maintenance). 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your 
class,  but  not  as 
good  as  the  top 
40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  safety  procedures 

Manages  power  sources  (e.g.,  performs  controlled  startup  and  shutdown  procedures,  ensure  equipment 
compatibility  with  power  sources  such  as  generators). 

Checks  for  proper  grounding. 

Removes  personal  jewelry  (e.g.,  watches,  rings)  before  working  on  equipment. 

Follows  safety  instructions  and  warnings. 

Identifies  trip  hazards  (e.g.,  unsecured  wires/cables). 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your 
class,  but  not  as 
good  as  the  top 
40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

A-2 


Table  A2 

25N  Training  Performance  Rating  Scales 


Learns  to  configure  devices 

•  Identifies  and  configures  equipment  (e.g.,  routers,  switches,  REDCOM,  PROMINA). 

•  Uses  NVRAM  and  ROM  appropriately  (e.g.  saves  information  to  a  flash  drive  or  compact  disc). 

•  Conducts  file  transfers  between  devices. 

1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your 
class,  but  not  as 
good  as  the  top 
40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  troubleshooting  techniques 

•  Performs  patching  (normal  patch  and  outer  normal  patch). 

•  Consults  technical  manual,  flow  charts  and/or  diagrams. 

•  Conducts  ping  tests  from  one  device  to  another  (e.g.,  computer  to  gateway  router). 

•  Inspects  and  services  hardware  and  software  (e.g.,  preventative  checks  and  maintenance). 

1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your  class, 
but  not  as  good  as  the 
top  40%) 

Above  Average 
(in  the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  to  maintain  communications  security  (COMSEC) 

•  Maintains  operation  security  (e.g.,  access  to  safes,  accounting  for  controlled  cryptographic  items  (CCI)). 

•  Maintains  transmission  security  (e.g.,  rules  for  encryption,  using  encryption  devices). 

•  Uses  various  types  of  encryption  devices  for  maintaining  security. 

1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your  class, 
but  not  as  good  as  the 
top  40%) 

Above  Average 
(in  the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  Internet  and  network  architecture 

•  Assigns  IP  addresses. 

•  Configures  LANs  and  WANs. 

•  Scales  a  network  using  sub-netting. 

•  Identifies  cables  required  for  connections  (e.g.,  serial,  Ethernet,  fiber-optic). 

1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average 
(in  the  bottom 
40%  of  Soldiers 
in  your  class) 

Average  (better  than  the 
bottom  40%  of  Soldiers 
in  your  class,  but  not  as 
good  as  the  top  40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

A-3 


Table  A2  (continued) 

25N  Training  Performance  Rating  Scales 


Learns  to  gain  access  to  devices  through  multiple  sources 

•  Accesses  a  terminal  server  (e.g.,  MRV). 

•  Uses  applications  to  access  devices  (e.g..  Pumpkin,  PUTTY). 

•  Configures  a  hyper  terminal. 

•  Accesses  the  KVM. 

•  Uses  a  URL  to  access  devices. 

•  Telnets  to  devices. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your  class, 
but  not  as  good  as  the 
top  40%) 

Above  Average 
(in  the  top  40% 
of  Soldiers  in 
your  class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  the  difference  between  TDMA  and  FDMA 


•  Identifies  the  types  of  devices  that  transmit  TDMA  and  FDMA. 

•  Identifies  the  signal  flow  through  the  appropriate  devices  (e.g.,  multiplexer,  TacLANe). 

•  Identifies  ports  for  TDMA  and  FDMA  devices. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your  class, 
but  not  as  good  as  the 
top  40%) 

Above  Average 
(in  the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  about  Internet  security 


•  Differentiates  among  various  malicious  attacks  (e.g.,  worms,  Trojan  horses,  phishing). 

•  Understands  the  purpose  of  a  firewall  and  brings  a  firewall  online. 

•  Uses  HTTP  and  HTTPS  appropriately. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average  (in 
the  bottom  40%  of 
Soldiers  in  your 
class) 

Average  (better  than  the 
bottom  40%  of  Soldiers 
in  your  class,  but  not  as 
good  as  the  top  40%) 

Above  Average 
(in  the  top  40% 
of  Soldiers  in 
your  class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

Learns  proper  safety  procedures 

•  Manages  power  sources  (e.g.,  controlled  startup  and  shutdown  procedures). 

•  Follows  safety  instructions  and  warnings. 

•  Checks  for  proper  grounding. 

•  Removes  personal  jewelry  (e.g.,  watches,  rings)  before  working  on  equipment. 


1 

2 

3 

4 

5 

Among  the  Weakest 
(in  the  bottom  20% 
of  Soldiers  in  your 
class) 

Below  Average 
(in  the  bottom 
40%  of  Soldiers  in 
your  class) 

Average  (better  than 
the  bottom  40%  of 
Soldiers  in  your  class, 
but  not  as  good  as  the 
top  40%) 

Above  Average  (in 
the  top  40%  of 
Soldiers  in  your 
class) 

Among  the 

Strongest  (in  the  top 
20%  of  Soldiers  in 
your  class) 

A-4 


Table  A3 

Summary  of  All  ALQ  Scale  Scores  by  MOS 


ALQ  Scale 

25B  (n 

=  1,012) 

25N  (n 

=  153) 

a 

M 

SD 

Min 

Max 

M 

SD 

Min 

Max 

Affective  Commitment 

.86 

3.8 

0.7 

1.0 

5.0 

3.7 

0.7 

1.6 

5.0 

Army  Civilian  Comparison 

.79 

3.7 

0.8 

1.0 

5.0 

3.8 

0.7 

1.2 

5.0 

Army  Fit 

.86 

4.0 

0.6 

1.0 

5.0 

3.9 

0.6 

2.1 

5.0 

Army  Life  Adjustment 

.88 

4.0 

0.7 

1.0 

5.0 

4.0 

0.6 

2.2 

5.0 

Attrition  Cognitions 

.75 

1.6 

0.6 

1.0 

4.0 

1.7 

0.7 

1.0 

4.8 

Career  Intentions 

.91 

3.1 

1.1 

1.0 

5.0 

2.9 

1.0 

1.0 

5.0 

Disciplinary  Actions 

- 

0.3 

0.7 

0.0 

5.0 

0.5 

0.9 

0.0 

7.0 

MOS  Fit 

.93 

3.8 

0.8 

1.1 

5.0 

3.6 

0.8 

1.1 

5.0 

Normative  Commitment 

.80 

4.0 

0.7 

1.0 

5.0 

4.0 

0.7 

1.4 

5.0 

Reenlistment  Intentions 

.81 

3.4 

0.9 

1.0 

5.0 

3.3 

0.9 

1.0 

5.0 

Training  Achievements 

- 

0.3 

0.5 

0.0 

2.0 

0.2 

0.4 

0.0 

1.0 

Training  Failures 

- 

0.5 

0.7 

0.0 

3.0 

0.5 

0.7 

0.0 

3.0 

*Coefficient  alpha  computed  using  25B  and  25N  samples  combined. 
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Table  A4 

Correlations  among  Predictor  and  Criterion  Variables  in  25B  MOS 


#  Variable 

n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

1  ICTL 

1.805 

2  El 

1,746 

.51 

3  AFQT 

1,746 

.55 

.46 

4  EL 

1,746 

.61 

.77 

.85 

5  SC 

1,746 

.61 

.74 

.88 

.99 

6  ST 

1,746 

.61 

.69 

.90 

.99 

.99 

7  Army  Civ.  Comparison 

1.004 

-.18 

-.13 

-.20 

-.21 

-.21 

-.22 

8  Affective  Commitment 

1,012 

-.07 

-.06 

-.10 

-.11 

-.11 

-.11 

.47 

9  Army  Life  Adjustment 

1,012 

.07 

.06 

.07 

.07 

.07 

.07 

.29 

.51 

10  Army  Fit 

1,012 

-.02 

-.02 

-.03 

-.05 

-.04 

-.05 

.44 

.83 

.67 

1 1  APFT 

995 

-.17 

-.13 

-.08 

-.13 

-.12 

-.13 

.03 

.06 

.20 

.10 

12  Attrition  Cognitions 

1,012 

.01 

-.02 

-.01 

-.01 

-.01 

-.01 

-.35 

-.60 

-.58 

-.66 

-.10 

13  Career  Intentions 

1,012 

-.18 

-.09 

-.17 

-.18 

-.18 

-.19 

.39 

.57 

.39 

.52 

.11 

-.52 

14  Disciplinary  Actions 

1,012 

-.01 

.05 

.03 

.04 

.04 

.04 

.01 

-.07 

-.12 

-.11 

-.07 

.04 

-.06 

15  MOS  Fit 

1,012 

.29 

.21 

.19 

.22 

.22 

.22 

.17 

.35 

.31 

.39 

-.11 

-.28 

.08 

-.02 

15  Normative  Commitment 

1,012 

.02 

.07 

.06 

.08 

.07 

.08 

.31 

.65 

.48 

.67 

.05 

-.71 

.46 

-.03 

.28 

17  Reenlistment  Intentions 

1,012 

-.15 

-.08 

-.15 

-.16 

-.15 

-.16 

.36 

.53 

.37 

.50 

.10 

-.53 

.86 

-.07 

.04 

.47 

1 8  Training  Achievements 

1,012 

-.13 

-.07 

-.12 

-.09 

-.10 

-.10 

.02 

.08 

.15 

.12 

.30 

-.10 

.10 

-.04 

.03 

.02 

.07 

19  Training  Failures 

1,012 

-.11 

-.07 

-.09 

-.13 

-.12 

-.12 

.04 

-.07 

-.18 

-.11 

-.25 

.10 

-.04 

.16 

-.08 

-.03 

-.03 

-.12 

20  WTBD  %  Correct 

959 

.36 

.32 

.44 

.44 

.44 

.45 

-.08 

.03 

.09 

.07 

-.02 

-.12 

-.06 

.00 

.10 

.14 

-.04 

-.05 

-.06 

21  PRS  Mean 

1,091 

.33 

.22 

.30 

.33 

.33 

.33 

-.08 

-.03 

.14 

.04 

.02 

-.01 

-.10 

-.11 

.23 

.01 

-.06 

.02 

-.12 

.20 

22  Final  AIT  Course  Grade 

527 

.40 

.31 

.39 

.45 

.46 

.46 

-.28 

-.15 

-.04 

-.09 

.01 

.09 

-.21 

.03 

.19 

.03 

-.17 

-.02 

-.08 

.46 

.40 

23  Grad  AIT  w/o  Failure 

1.442 

.16 

.05 

.16 

.15 

.16 

.16 

.02 

-.01 

.06 

.03 

.08 

-.05 

-.03 

-.11 

.05 

-.01 

-.03 

.01 

-.37 

.04 

.20 

.33 

Note.  For  all  correlations,  bolded  values  are  those  where  p  <  0.01  and  italicized  values  are  those  where  p  <  0.05. 
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Table  A5 

Correlations  among  Predictor  and  Criterion  Variables  in  25N  MOS 


#  Variable 

n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

1  ICTL 

314 

2  El 

294 

.36 

3  AFQT 

294 

.38 

.26 

4  EL 

294 

.44 

.66 

.79 

5  SC 

294 

.43 

.64 

.83 

.99 

6  ST 

294 

.44 

.56 

.87 

.98 

.99 

7  Army  Civ.  Comparison 

151 

-.13 

-.14 

.02 

-.14 

-.12 

-.11 

8  Affective  Commitment 

153 

-.05 

-.11 

-.01 

-.15 

-.13 

-.11 

.52 

9  Army  Life  Adjustment 

153 

-.14 

-.19 

.02 

-.12 

-.10 

-.09 

.22 

.58 

10  Army  Fit 

153 

-.06 

-.16 

.03 

-.12 

-.10 

-.09 

.42 

.85 

.71 

11  APFT 

153 

-.20 

-.20 

-.03 

-.14 

-.11 

-.12 

-.03 

.08 

.26 

.13 

12  Attrition  Cognitions 

153 

.16 

.16 

-.04 

.10 

.08 

.07 

-.39 

-.68 

-.61 

-.72 

-.17 

13  Career  Intentions 

153 

-.11 

-.06 

-.20 

-.19 

-.19 

-.19 

.33 

.54 

.38 

.46 

.02 

-.43 

14  Disciplinary  Actions 

153 

-.02 

.01 

-.03 

-.05 

-.06 

-.05 

.00 

-.16 

-.25 

-.19 

-.22 

.28 

-.18 

15  MOS  Fit 

153 

.16 

.06 

.05 

.00 

.01 

.01 

.43 

.39 

.24 

.34 

-.10 

-.27 

.14 

.04 

16  Normative  Commitment 

153 

-.10 

-.11 

.07 

-.03 

-.01 

.00 

.46 

.74 

.54 

.75 

-.03 

-.82 

.47 

-.27 

.33 

17  Reenlistment  Intentions 

153 

-.19 

-.10 

-.15 

-.18 

-.18 

-.18 

.34 

.52 

.45 

.50 

.04 

-.47 

.85 

-.15 

.11 

.48 

18  Training  Achievements 

153 

-.20 

-.12 

-.31 

-.24 

-.24 

-.26 

-.05 

-.02 

.06 

-.01 

.32 

-.09 

.08 

-.04 

-.11 

.00 

.13 

19  Training  Failures 

153 

.16 

.23 

.08 

.12 

.12 

.11 

.05 

-.02 

-.20 

-.12 

-.34 

.20 

-.12 

.26 

.15 

-.08 

-.14 

-.32 

20  WTBD  %  Correct 

146 

.18 

.05 

.18 

.13 

.13 

.13 

.03 

.06 

.18 

.12 

-.08 

.01 

-.08 

.11 

.13 

.10 

-.07 

-.05 

.07 

21  PRS  Mean 

169 

.14 

.12 

.03 

.08 

.09 

.07 

-.16 

-.07 

.08 

.00 

.19 

-.07 

-.10 

-.18 

.06 

.03 

-.14 

.19 

-.12 

.09 

22  Final  AIT  Course  Grade 

159 

.46 

.26 

.45 

.45 

.47 

.47 

-.14 

-.11 

.11 

-.03 

.18 

-.08 

-.15 

-.20 

-.07 

.00 

-.15 

.00 

-.14 

.27 

.42 

23  Grad  AIT  w/o  Failure 

229 

.08 

.04 

.11 

.08 

.10 

.10 

.38 

.23 

.22 

.30 

.21 

-.37 

.13 

-.24 

.13 

.32 

.16 

.14 

-.20 

.07 

.05 

.24 

Note.  For  all  correlations,  bolded  values  are  those  where  p  <  0.01  and  italicized  values  are  those  where  p  <  0.05. 
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Table  A6 

Gender  Subgroup  Means,  Standard  Deviations,  and  Standardized  Group  Mean  Differences  ( Cohen ’s  d)  among  Predictor  Variables  in 
25 B  and  25N  MOS 


25  B 

25N 

Female 

Male 

Female 

Male 

Measure 

M 

SD 

M 

SD 

d 

M 

SD 

M 

SD 

d 

ICTL  Scaled  Score 

51.96 

6.38 

56.12 

8.28 

-0.52 

57.97 

7.69 

60.52 

7.57 

-0.34 

ASVAB:  El  Subtest 

46.65 

6.55 

53.39 

8.34 

-0.84 

52.38 

5.34 

57.24 

7.18 

-0.70 

ASVAB:  Skilled -Technical  Composite 

102.46 

7.36 

108.96 

11.18 

-0.62 

112.23 

7.18 

116.02 

8.73 

-0.44 

ASVAB:  Electrical  Composite 

100.55 

7.54 

108.38 

11.63 

-0.72 

110.82 

7.26 

115.68 

8.90 

-0.56 

ASVAB:  Surveillance  and  Communication 
Composite 

102.34 

7.49 

109.18 

11.29 

-0.65 

112.36 

6.96 

116.33 

8.69 

-0.47 

Note.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  subgroup  sample  size  is  1,371  and  359  for  Males  and  Females,  respectively. 
For  25N,  subgroup  sample  size  is  254  and  39  for  Males  and  Females,  respectively.  Significant  mean  differences,  based  on  an  independent  sample  f-test 
between  the  group  means,  are  denoted  by  a  d  value  in  bold  typeface  (p  <  .05,  two-tailed). 
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Table  A7 

Racial/Ethnic  Subgroup  Means,  Standard  Deviations,  and  Standardized  Group  Mean  Differences  ( Cohen ’s  d)  among  Predictor 
Variables  in  25 B  and  25N  MOS 


White 

Black 

Hispanic 

MOS  /  Measure 

M 

SD 

M 

SD 

d 

M 

SD 

d 

25B 

ICTL  Scaled  Score 

57.43 

8.35 

52.52 

7.05 

-0.62 

53.98 

7.38 

-0.42 

ASVAB:  El  Subtest 

54.89 

7.92 

48.85 

7.66 

-0.77 

50.97 

8.19 

-0.49 

ASVAB:  Skilled-Technical  Composite 

111.98 

11.50 

102.76 

7.56 

-0.91 

104.61 

8.86 

-0.67 

ASVAB:  Electrical  Composite 

111.51 

11.92 

101.41 

7.97 

-0.96 

103.86 

9.29 

-0.67 

ASVAB:  Surveillance  and  Communication 

112.08 

11.65 

102.91 

7.74 

-0.89 

104.96 

9.03 

-0.64 

Composite 

25N 

ICTL  Scaled  Score 

61.01 

7.69 

57.00 

7.52 

-0.53 

59.57 

6.84 

-0.19 

ASVAB:  El  Subtest 

57.57 

6.74 

54.53 

7.50 

-0.44 

54.78 

6.91 

-0.41 

ASVAB:  Skilled-Technical  Composite 

117.66 

8.87 

111.27 

6.65 

-0.77 

112.43 

7.23 

-0.61 

ASVAB:  Electrical  Composite 

117.26 

8.98 

110.61 

6.93 

-0.79 

111.68 

7.36 

-0.64 

ASVAB:  Surveillance  and  Communication 

117.77 

8.90 

111.82 

6.50 

-0.72 

112.89 

7.13 

-0.57 

Composite 

Note.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  subgroup  sample  sizes  are  522,  247,  and  803  for  Blacks, 
Hispanics,  and  Whites,  respectively.  For  25N,  subgroup  sample  sizes  are  62,  37,  and  164  for  Blacks,  Hispanics,  and  Whites,  respectively. 
Significant  mean  differences,  based  on  an  independent  sample  f- test  between  the  group  means,  are  denoted  by  a  cl  value  in  bold  typeface  (p 
<  .05,  two-tailed). 
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Table  A8 

Gender  Subgroup  Means,  Standard  Deviations,  and  Standardized  Group  Mean  Differences  ( Cohen ’s  d)  among  Criterion  Variables  in 
25 B  and  25N  MOS 


Measure 

25  B 

25  N 

Female 

Male 

d 

Female 

Male 

d 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

Final  AIT  course  grade 

81.00 

9.20 

81.95 

9.76 

-0.10 

91.41 

4.88 

92.55 

5.05 

-0.23 

AIT  graduation  status 

0.87 

0.34 

0.86 

0.35 

0.05 

0.97 

0.19 

0.90 

0.30 

0.23 

Peer  performance  rating  scales  (PRS) 

3.78 

0.75 

4.03 

0.79 

-0.32 

3.91 

0.85 

4.03 

0.59 

-0.19 

MOS  fit 

3.51 

0.80 

3.94 

0.74 

-0.57 

3.09 

1.17 

3.63 

0.78 

-0.64 

Note.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  sample  sizes  for  Males  range  from  413  to  1,132;  Females  from 
1 1 1  to  303.  For  25N,  sample  sizes  for  Males  range  from  125  to  199;  Females  from  18  to  29.  Significant  mean  differences,  based  on  an 
independent  sample  f-test  between  the  group  means,  are  denoted  by  a  d  value  in  bold  typeface  (p  <  .05,  two-tailed). 
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Table  A9 

Racial/Ethnic  Subgroup  Means,  Standard  Deviations,  and  Standardized  Group  Mean  Differences  ( Cohen ’s  d)  among  Criterion 
Variables  in  25 B  and  25N  MOS 


White 

Black 

Hispanic 

MOS  /  Measure 

M 

SD 

M 

SD 

d 

M 

SD 

d 

25B 

Final  AIT  course  grade 

83.36 

9.07 

78.95 

10.49 

-0.46 

81.80 

7.60 

-0.18 

AIT  graduation  status 

0.88 

0.33 

0.82 

0.39 

-0.16 

0.86 

0.34 

-0.04 

Peer  performance  rating  scales  (PRS) 

4.08 

0.77 

3.84 

0.82 

-0.30 

3.90 

0.76 

-0.23 

MOS  fit 

3.94 

0.80 

3.78 

0.72 

-0.21 

3.81 

0.71 

-0.17 

25N 

Final  AIT  course  grade 

92.76 

4.94 

90.45 

5.75 

-0.45 

93.53 

2.89 

0.16 

AIT  graduation  status 

0.91 

0.29 

0.94 

0.24 

0.11 

0.90 

0.31 

-0.04 

Peer  performance  rating  scales  (PRS) 

3.97 

0.65 

4.09 

0.69 

0.18 

3.98 

0.47 

0.01 

MOS  fit 

3.42 

0.90 

3.95 

0.60 

0.65 

3.29 

0.92 

-0.14 

Note.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  sample  sizes  for  White,  Non-Hispanics  range  from  245  to 
642;  Blacks  from  157  to  442;  Hispanics  from  71  to  219.  For  25N,  sample  sizes  for  White,  Non-Hispanics  range  from  82  to  130;  Blacks 
from  27  to  48;  Hispanics  from  17  to  29.  Significant  mean  differences,  based  on  an  independent  sample  f-test  between  the  group  means, 
are  denoted  by  a  d  value  in  bold  typeface  (p  <  .05,  two-tailed). 
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Table  A10 

Gender  Subgroup  Pass  Rates  and  Adverse  Impact  Statistics  for  Various  Cutoffs  on  the  ICTL  Test  in  25B  and  25N  MOS 


25  B 

25N 

Measure  /  Cutoff 

Female 

PR 

Male 

PR 

AIR 

FET 

Female 

PR 

Male 

PR 

AIR 

FET 

ICTL  Scaled  Score 

50th  Percentile 

28.4% 

50.8% 

.56 

.00 

41.0% 

51.2% 

.80 

.30 

55th  Percentile 

24.5% 

46.5% 

.53 

.00 

35.9% 

44.5% 

.81 

.39 

60th  Percentile 

19.5% 

41.6% 

.47 

.00 

33.3% 

40.9% 

.81 

.39 

65th  Percentile 

15.0% 

37.3% 

.40 

.00 

23.1% 

35.8% 

.64 

.15 

70th  Percentile 

12.0% 

34.1% 

.35 

.00 

17.9% 

32.7% 

.55 

.09 

Note.  PR  =  Pass  rate;  AIR  =  adverse  impact  ratio;  FET  =  Fisher's  exact  test.  The  minority  group  is  the  referent  group  for  the 
comparisons.  For  25B,  Male  sample  size  is  1,371;  female  sample  size  is  359.  For  25N,  Male  sample  size  is  254;  female  sample  size 
is  39.  Fisher's  exact  test  values  <  0.05  highlighted  in  bold  typeface. 
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Table  All 

Racial/Ethnic  Subgroup  Pass  Rates  and  Adverse  Impact  Statistics  for  Various  Cutoffs  on  the  ICTL  Test  in  25B  and  25N  MOS 


25B  25N 


Measure  /  Cutoff 

White 

PR 

Black 

PR 

AIR 

FET 

Hisp 

PR 

AIR 

FET 

White 

PR 

Black 

PR 

AIR 

FET 

Hisp 

PR 

AIR 

FET 

ICTL  Scaled  Score 

50th  Percentile 

57.5% 

32.0% 

.56 

.00 

40.5% 

.70 

.00 

53.7% 

33.9% 

.63 

.01 

40.5% 

.76 

.20 

55th  Percentile 

53.7% 

27.4% 

.51 

.00 

35.6% 

.66 

.00 

48.2% 

24.2% 

.50 

.00 

37.8% 

.79 

.28 

60th  Percentile 

48.6% 

22.2% 

.46 

.00 

31.2% 

.64 

.00 

45.7% 

19.4% 

.42 

.00 

32.4% 

.71 

.15 

65th  Percentile 

44.0% 

18.2% 

.41 

.00 

28.7% 

.65 

.00 

39.0% 

17.7% 

.45 

.00 

24.3% 

.62 

.13 

70th  Percentile 

41.0% 

15.5% 

.38 

.00 

23.5% 

.57 

.00 

35.4% 

12.9% 

.36 

.00 

24.3% 

.69 

.25 

Note.  PR  =  Pass  rate;  AIR  =  adverse  impact  ratio;  FET  =  Fisher's  exact  test.  The  minority  group  is  the  referent  group  for  the  comparisons.  For  25B,  White 
sample  size  is  803;  Black  sample  size  is  522;  Hispanic  sample  size  is  247.  For  25N,  White  sample  size  is  164;  Black  sample  size  is  62;  Hispanic  sample  size  is 
37.  Fisher's  exact  test  values  <  0.05  highlighted  in  bold  typeface. 
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Table  A12 

Gender  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for  Differential  Prediction 
Analyses 


Correlations  Moderated  Multiple  Regression 

Male  Female  Uncorrected  Corrected 

Criterion  /  Predictor _ r  rRRC _ r  rRRC _ Int  X  Grp  X*Grp _ Int  X  Grp  X*Grp 

Final  AIT  course  grade 


ICTL  Scaled  Score 

.433 

.501 

.276 

ASVAB:  El  Subtest 

.357 

.411 

.100 

ASVAB:  Skilled -Technical  Composite 

.490 

.562 

.282 

ASVAB:  Electrical  Composite 

.486 

.557 

.287 

ASVAB:  Surveillance  and  Communication 
Composite 

.495 

.567 

.293 

AIT  graduation  status 


ICTL  Scaled  Score 

.174 

.226 

.126 

ASVAB:  El  Subtest 

.068 

.129 

.008 

ASVAB:  Skilled -Technical  Composite 

.174 

.235 

.137 

ASVAB:  Electrical  Composite 

.172 

.235 

.127 

ASVAB:  Surveillance  and  Communication 
Composite 

.177 

.240 

.142 

Peer  performance  rating  scales  (PRS) 


ICTL  Scaled  Score 

.325 

.408 

.232 

ASVAB:  El  Subtest 

.222 

.306 

.011 

ASVAB:  Skilled-Technical  Composite 

.342 

.438 

.133 

ASVAB:  Electrical  Composite 

.337 

.433 

.124 

ASVAB:  Surveillance  and  Communication 
Composite 

.344 

.440 

.115 

.426 

81.38 

0.51 

0.86 

-0.12 

80.99 

0.57 

0.57 

-0.01 

.309 

81.39 

0.41 

0.30 

-0.28 

79.66 

0.47 

1.20 

-0.08 

.486 

81.44 

0.41 

0.83 

-0.08 

81.46 

0.39 

1.04 

-0.03 

.483 

81.33 

0.39 

1.30 

-0.07 

81.15 

0.38 

1.62 

-0.02 

.487 

81.41 

0.41 

0.99 

-0.08 

81.47 

0.39 

1.14 

-0.03 

.227 

0.85 

0.01 

0.04 

0.00 

0.83 

0.01 

0.04 

0.00 

.115 

0.85 

0.00 

0.02 

0.00 

0.82 

0.01 

0.03 

0.00 

.271 

0.85 

0.01 

0.06 

0.00 

0.84 

0.01 

0.06 

0.00 

.263 

0.85 

0.01 

0.06 

0.00 

0.84 

0.01 

0.06 

0.00 

.274 

0.85 

0.01 

0.06 

0.00 

0.84 

0.01 

0.06 

0.00 

.302 

3.98 

0.03 

-0.12 

0.00 

3.94 

0.04 

-0.07 

-0.01 

.131 

4.00 

0.02 

-0.20 

-0.02 

3.86 

0.03 

-0.07 

-0.01 

.253 

3.99 

0.02 

-0.13 

-0.01 

3.97 

0.02 

-0.10 

-0.01 

.248 

3.98 

0.02 

-0.12 

-0.01 

3.95 

0.02 

-0.07 

-0.01 

.244 

3.99 

0.02 

-0.14 

-0.01 

3.97 

0.02 

-0.10 

-0.01 

(continued) 
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Table  A12  (continued) 

Gender  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for  Differential  Prediction 
Analyses 


Correlations  Moderated  Multiple  Regression 

Male  Female  Uncorrected  Corrected 


Criterion  /  Predictor 

r 

rRRc 

r 

rRRc 

Int 

X 

Grp 

X*Grp 

Int 

X 

Grp 

X*Grp 

M OS  fit 

ICTL  Scaled  Score 

.263 

.301 

.226 

.337 

3.90 

0.02 

-0.30 

0.00 

3.90 

0.02 

-0.29 

0.01 

ASVAB;  El  Subtest 

.157 

.200 

.180 

.290 

3.91 

0.01 

-0.28 

0.01 

3.84 

0.02 

-0.25 

0.01 

ASVAB:  Skilled-Technical  Composite 

.182 

.237 

.174 

.323 

3.91 

0.01 

-0.31 

0.01 

3.90 

0.01 

-0.27 

0.01 

ASVAB:  Electrical  Composite 

.178 

.233 

.201 

.333 

3.91 

0.01 

-0.27 

0.01 

3.89 

0.01 

-0.23 

0.01 

ASVAB:  Surveillance  and  Communication 
Composite 

.183 

.236 

.188 

.325 

3.91 

0.01 

-0.30 

0.01 

3.90 

0.01 

-0.26 

0.01 

Note.  Uncorrected  estimates  based  on  the  validation  sample;  corrected  estimates  based  on  range  restriction  corrections  as  explained  in  text.  Moderated  multiple 
regression  results  are  unstandardized  coefficients  for  a  model  that  includes  the  focal  predictor  measure  (X),  demographic  group  variable  (Grp;  coded  as  0  = 


Majority,  1  =  Minority),  and  the  interaction  between  the  focal  predictor  and  the  demographic  group  variable  (X*Grp).  Sample  sizes  for  Males  range  from  413  to 
1,132;  for  Females,  from  1 1 1  to  303.  For  all  tabled  values,  bolded  values  are  those  where  p  <  0.05.  Italicized  correlations  are  those  where  the  difference  between 
the  Majority  and  Minority  estimates  is  significant  at  two-tailed  p  <  0.05. 
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Table  A13 

Racial/Ethnic  (Hispanic/White)  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for 
Differential  Prediction  Analyses 


Correlations  Moderated  Multiple  Regression 


White  Hispanic  Uncorrected  Corrected 

Criterion  /  Predictor _ r  rRRC _ r  rRRC _ Int  X  Grp  X*Grp _ Int  X  Grp  X*Grp 

Final  AIT  course  grade 


ICTL  Scaled  Score 

.415 

.464 

.288 

.357 

82.23 

0.45 

-0.15 

-0.17 

81.72 

0.50 

0.07 

-0.17 

ASVAB:  El  Subtest 

.342 

.370 

.077 

.173 

82.33 

0.37 

-0.50 

-0.30 

80.60 

0.42 

0.23 

-0.27 

ASVAB:  Skilled-Technical  Composite 

.520 

.571 

.277 

.346 

81.59 

0.39 

0.76 

-0.18 

81.73 

0.38 

0.63 

-0.17 

ASVAB:  Electrical  Composite 

.511 

.561 

.263 

.339 

81.53 

0.37 

0.75 

-0.17 

81.42 

0.37 

0.83 

-0.17 

ASVAB:  Surveillance  and  Communication 
Composite 

AIT  graduation  status 

.517 

.570 

.271 

.346 

81.66 

0.38 

0.62 

-0.17 

81.77 

0.38 

0.57 

-0.17 

ICTL  Scaled  Score 

.142 

.167 

.099 

.094 

0.86 

0.01 

0.01 

0.00 

0.85 

0.01 

0.01 

0.00 

ASVAB:  El  Subtest 

.075 

.094 

-.024 

.003 

0.87 

0.00 

0.00 

0.00 

0.84 

0.00 

0.01 

0.00 

ASVAB:  Skilled-Technical  Composite 

.190 

.216 

.034 

.035 

0.85 

0.01 

0.01 

0.00 

0.85 

0.01 

0.00 

0.00 

ASVAB:  Electrical  Composite 

.185 

.212 

.032 

.038 

0.85 

0.01 

0.01 

0.00 

0.85 

0.00 

0.01 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

Peer  performance  rating  scales  (PRS) 

.189 

.217 

.042 

.043 

0.85 

0.01 

0.01 

0.00 

0.85 

0.01 

0.01 

0.00 

ICTL  Scaled  Score 

.340 

.398 

.262 

.313 

3.98 

0.03 

-0.06 

0.00 

3.96 

0.04 

-0.05 

0.00 

ASVAB:  El  Subtest 

.235 

.273 

.120 

.188 

4.01 

0.02 

-0.09 

-0.01 

3.89 

0.03 

-0.06 

-0.01 

ASVAB:  Skilled-Technical  Composite 

.357 

.423 

.221 

.275 

3.96 

0.02 

0.00 

0.00 

3.96 

0.02 

-0.02 

-0.01 

ASVAB:  Electrical  Composite 

.349 

.413 

.238 

.292 

3.96 

0.02 

0.00 

0.00 

3.94 

0.02 

0.00 

-0.01 

ASVAB:  Surveillance  and  Communication 
Composite 

.354 

.420 

.229 

.281 

3.97 

0.02 

-0.01 

0.00 

3.97 

0.02 

-0.02  -0.01 

(continued) 
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Table  A13  (continued) 

Racial/Ethnic  (Hispanic/White)  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for 
Differential  Prediction  Analyses 


Correlations  Moderated  Multiple  Regression 


White  Hispanic  Uncorrected  Corrected 


Criterion  /  Predictor 

r 

rRRc 

r 

I'RRC 

Int 

X 

Grp 

X*Grp 

Int 

X 

Grp 

X*Grp 

MOSfit 

ICTL  Scaled  Score 

.338 

.370 

.147 

.158 

3.85 

0.03 

-0.03 

-0.02 

3.84 

0.03 

0.00 

-0.02 

ASVAB;  El  Subtest 

.258 

.295 

.057 

.074 

3.86 

0.03 

-0.05 

-0.02 

3.77 

0.03 

0.04 

-0.02 

ASVAB:  Skilled-Technical  Composite 

.266 

.308 

.020 

.068 

3.85 

0.02 

-0.04 

-0.02 

3.85 

0.02 

-0.03 

-0.01 

ASVAB:  Electrical  Composite 

.270 

.314 

.016 

.057 

3.84 

0.02 

-0.03 

-0.02 

3.83 

0.02 

-0.02 

-0.01 

ASVAB:  Surveillance  and  Communication 
Composite 

.264 

.306 

.024 

.065 

3.85 

0.02 

-0.04 

-0.02 

3.85 

0.02 

-0.03 

-0.01 

Note.  Uncorrected  estimates  based  on  the  validation  sample;  corrected  estimates  based  on  range  restriction  corrections  as  explained  in  text.  Moderated  multiple 
regression  results  are  unstandardized  coefficients  for  a  model  that  includes  the  focal  predictor  measure  (X),  demographic  group  variable  (Grp;  coded  as  0  = 
Majority,  1  =  Minority),  and  the  interaction  between  the  focal  predictor  and  the  demographic  group  variable  (X*Grp).  Sample  sizes  for  Whites  range  from  245  to 
642;  for  Hispanics,  from  71  to  219.  For  all  tabled  values,  bolded  values  are  those  where  p  <  0.05.  Italicized  correlations  are  those  where  the  difference  between 
the  Majority  and  Minority  estimates  is  significant  at  two-tailed  p  <  0.05. 
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Table  A14 

Racial/Ethnic  (BlackAVhite)  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for 
Differential  Prediction  Analyses 


Correlations  Moderated  Multiple  Regression 

White  Black  Uncorrected  Corrected 


Criterion  /  Predictor 

r 

rRRc 

r 

I'RRC 

Int 

X 

Grp 

X*Grp 

Int 

X 

Grp 

X*Grp 

Final  AIT  course  grade 

ICTL  Scaled  Score 

.415 

.464 

.319 

.462 

82.23 

0.45 

-1.92 

0.06 

81.72 

0.50 

-2.68 

0.24 

ASVAB:  El  Subtest 

.342 

.370 

.224 

.389 

82.33 

0.37 

-2.37 

-0.05 

80.60 

0.42 

-3.40 

0.14 

ASVAB:  Skilled-Technical  Composite 

.520 

.571 

.365 

.560 

81.59 

0.39 

-0.26 

0.12 

81.73 

0.38 

-0.33 

0.15 

ASVAB:  Electrical  Composite 

.511 

.561 

.360 

.556 

81.53 

0.37 

-0.14 

0.11 

81.42 

0.37 

0.05 

0.15 

ASVAB:  Surveillance  and  Communication 

.517 

.570 

.390 

.574 

81.66 

0.38 

-0.23 

0.15 

81.77 

0.38 

-0.24 

0.16 

Composite 


AIT  graduation  status 


ICTL  Scaled  Score 

.142 

.167 

.210 

.298 

0.86 

0.01 

-0.01 

0.01 

0.85 

0.01 

-0.02 

0.01 

ASVAB:  El  Subtest 

.075 

.094 

-.001 

.108 

0.87 

0.00 

-0.05 

0.00 

0.84 

0.00 

-0.08 

0.00 

ASVAB:  Skilled-Technical  Composite 

.190 

.216 

.144 

.283 

0.85 

0.01 

0.00 

0.00 

0.85 

0.01 

0.01 

0.00 

ASVAB:  Electrical  Composite 

.185 

.212 

.137 

.278 

0.85 

0.01 

0.00 

0.00 

0.85 

0.00 

0.01 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

Peer  performance  rating  scales  (PRS) 

.189 

.217 

.148 

.287 

0.85 

0.01 

0.00 

0.00 

0.85 

0.01 

0.01 

0.00 

ICTL  Scaled  Score 

.340 

.398 

.227 

.334 

3.98 

0.03 

-0.09 

-0.01 

3.96 

0.04 

-0.12 

0.00 

ASVAB:  El  Subtest 

.235 

.273 

.102 

.208 

4.01 

0.02 

-0.14 

-0.01 

3.89 

0.03 

-0.17 

0.00 

ASVAB:  Skilled-Technical  Composite 

.357 

.423 

.210 

.351 

3.96 

0.02 

-0.03 

0.00 

3.96 

0.02 

-0.03 

0.00 

ASVAB:  Electrical  Composite 

.349 

.413 

.200 

.344 

3.96 

0.02 

-0.03 

0.00 

3.94 

0.02 

-0.01 

0.00 

ASVAB:  Surveillance  and  Communication 
Composite 

.354 

.420 

.204 

.348 

3.97 

0.02 

-0.04 

0.00 

3.97 

0.02 

-0.03  0.00 

(continued) 
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Table  A 14  (continued) 

Racial/Ethnic  (Black/White)  Subgroup  Criterion-related  Validity  Estimates  and  Moderated  Multiple  Regression  Results  for 
Differential  Prediction  Analyses 


Correlations  Moderated  Multiple  Regression 

White  Black  Uncorrected  Corrected 


Criterion  /  Predictor 

r 

rRRc 

r 

I'RRC 

Int 

X 

Grp 

X*Grp 

Int 

X 

Grp 

X*Grp 

MOSfit 

ICTL  Scaled  Score 

.338 

.370 

.208 

.268 

3.85 

0.03 

-0.03 

-0.01 

3.84 

0.03 

-0.06 

-0.01 

ASVAB:  El  Subtest 

.258 

.295 

.138 

.195 

3.86 

0.03 

-0.05 

-0.01 

3.77 

0.03 

-0.04 

-0.01 

ASVAB:  Skilled-Technical  Composite 

.266 

.308 

.136 

.228 

3.85 

0.02 

-0.01 

-0.01 

3.85 

0.02 

-0.04 

0.00 

ASVAB:  Electrical  Composite 

.270 

.314 

.148 

.236 

3.84 

0.02 

0.00 

-0.01 

3.83 

0.02 

-0.01 

0.00 

ASVAB:  Surveillance  and  Communication 

.264 

.306 

.156 

.241 

3.85 

0.02 

-0.01 

0.00 

3.85 

0.02 

-0.03 

0.00 

Composite 

Note.  Uncorrected  estimates  based  on  the  validation  sample;  corrected  estimates  based  on  range  restriction  corrections  as  explained  in  text.  Moderated  multiple 
regression  results  are  unstandardized  coefficients  for  a  model  that  includes  the  focal  predictor  measure  (X),  demographic  group  variable  (Grp;  coded  as  0  = 
Majority,  1  =  Minority),  and  the  interaction  between  the  focal  predictor  and  the  demographic  group  variable  (X*Grp).  Sample  sizes  for  Whites  range  from  245 
to  642;  for  Blacks,  from  157  to  442.  For  all  tabled  values,  bolded  values  are  those  where  p  <  0.05.  Italicized  correlations  are  those  where  the  difference  between 
the  Majority  and  Minority  estimates  is  significant  at  two-tailed  p  <  0.05. 
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